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PREPACK 


A  paper  entitled  "A  revised  pilot  rating  scale  for  the  evaluation  of 
handling  qualities"  by  G.E. Cooper  and  R.  P. Harper,  presented  at  the 
Stability  and  Control  meeting  organised  by  the  AGARD  Flight  Mechanics 
Panel  in  Cambridge,  England  in  September  J £>66,  was  considered  by  the 
Panel  to  offer  a  good  basis  for  the  production  of  a  standard  Pilot’s 
Handling  Rating  Scale.  The  present  paper  is  a  development  by  the 
authors  of  the  earlier  paper,  including  additional  explanatory 
discussion,  and  taking  account  of  views  and  comments  of  other  users  of 
rating  scales  in  a  rojnber  of  NATO  countries.  The  form  of  scale,  and 
tht  method  of  its  application  proposed  in  this  paper  have  now  been 
widel;*  accepted  as  representing  a  suitable  basis  for  a  standard 
rating  scale.  It  has  the  afore  been  adopted  by  the  Flight  Mechanics 
Panel  as  the  standard  AGARD  Rating  Scale,  and  its  use  is  recommended, 
to  provide  greater  uniformity  of  rating  assessment  of  aircraft 
handling  qualities. 


Tin;  USE  OF  PILOT  RATING  IN  THE  EVALUATION  OF 


AIRCRAFT  HANDLING  QUALITIES 
By  George  E.  Cooper 
Ames  Research  Center 
and 

Robert  P.  Harper,  Jr. 
Cornell  Aeronautical  Laboratory 

SUMMARY 


Pilot  rating  scales  and  their  use  in  assessing  aircraft  handling 
qualities  are  reviewed  historically,  and  objections  that  have  been  raised  to 
limitations  of  earlier  scales  are  considered  in  the  development  of  a  revised 
scale.  Terminology  used  in  the  evaluation  of  handling  qualities  is  reviewed 
and  new  definitions  are  proposed  to  improve  communication  and  international 
understanding.  Of  particular  significance  is  the  new  definition  of  handling 
qualities,  which  emphasizes  the  importance  of  factors  that  influence  the  selec 
tion  of  a  rating  other  than  stability  and  control  characteristics. 

The  experimental  use  of  pilot  rating  is  discussed  in  detail,  with  special 
attention  devoted  to  (1)  clarifying  the  difference  between  mission  and  task, 

(2 J  identifying  what  the  rating  applies  to,  (3)  considering  the  pilot's 
assessment  criteria,  and  (4)  defining  the  simulation  situation.  Tne  important 
elements  of  the  report  are  then  summarized  in  a  suggested  "Briefing  Guide," 
designed  for  guidance  in  planning  and  executing  handling  qualities 
experiments . 


INTRODUCTION 


The  widespread  application  of  pilot  rating  scales  in  the  evaluation  of 
aircraft  handling  qualities  has  confirmed  their  basic  utility,  but  has,  at  the 
same  time,  exposed  some  weaknesses  of  the  scales  as  originally  proposed.  It 
was  therefore  considered  desirable  to  re-examine  existing  rating  scales  with 
the  purpose  of  developing  a  single  improved  scale  and  of  clarifying  its  use  i  ri 
the  evaluation  of  handling  qualities.  In  response  to  an  invitation  from  the 
E light  Mechanics  Panel ,  AGARD,  a  paper  entitled,  "A  Revised  Pilot  Rating  Scale 
for  the  Evaluation  of  Handling  Qualities,”  was  prepared  and  presented  at  the 
September  i960  meeting  (ref.  1).  A  longer  version  of  this  paper  (ref .  2)  con¬ 
tains  some  additional  explanatory  discussion  of  the  use  of  pi  lot  rating  scales 
Additional  constructive  criticism  was  then  based  on  experience  gained  with 
this  revised  scale. 


In  general,  the  revised  scale  was  preferred  over  earlier  sca.es,  but 
constructive  criticism  from  many  research  and  development  groups  also 
indicated  the  need  for  additional  changes  or  clarifying  discussion.  Diffi¬ 
culties,  for  example,  were  experienced  with  the  semantics  in  that  certain 
words  had  rather  different  connotations  in  the  United  States,  England,  and  in 
France.  The  purpose  of  this  report,  then,  is  to  clarify  and  modify,  as  appro¬ 
priate,  the  material  presented  in  references  1  and  2.  One  of  the  first  objec¬ 
tives  of  the  present  report  is  to  define  precisely  the  basic  terminology  and 
explain  the  new  features  in  the  scale.  The  report  goes  on  to  discuss  the  more 
important  factors  that  are  considered  by  the  pilot  in  the  selection  of  a  rat¬ 
ing  or  that  will  otherwise  influence  the  rating.  Throughout  the  discussion, 
attention  is  directed  to  the  questions:  (1)  what  is  the  pilot  being  asked  to 
rate?,  and  (2)  how  will  the  experimental  results  be  used?  The  answers  to  both 
questions  have  important  bearing  on  the  interpretation  of  evaluations  made  by 
pilots  with  different  backgrounds,  experience,  and  points  of  view.  The  final 
section  of  the  discussion  is  devoted  to  a  review  of  certain  other  consider¬ 
ations  that  are  helpful  in  the  design  of  handling  qualities  experiments  and  to 
the  use  of  pilot  rating.  The  important  elements  of  the  report  are  summarized 
in  a  condensed  "Briefing  Guide"  for  use  in  planning  and  executing  handling 
qualities  experiments. 


DISCUSSION  OF  HANDLING  QUALITIES 


Clarification  of  Terms 

For  a  pilot  rating  scale  to  be  universally  acceptable  and  consistently 
applied  in  the  evaluation  of  handling  qualities,  the  terminology  must  be 
easily  understood  by  all  persons  working  in  the  field.  Those  terms  requiring 
specific  attention  are  defined  in  appendix  A,  and  several  definitions  are  sug¬ 
gested  that  may  help  clarify  and  standardize  the  terminology.  Some  of  the 
terms  suggested  in  reference  3  have  been  adopted  in  this  report.  Those  most 
significant  to  a  discussion  of  handling  qualities  are  examined  in  considerable 
detail  in  the  following  paragraphs.  Others  are  discussed  as  the  need  arises. 

Handling  qualities.-  The  term  "Handling  Qualities"  requires  a  clear 
definition  in  order  to  emphasize  that  it  includes  more  than  just  stability  and 
control  characteristics.  Other  factors  that  influence  the  handling  qualities 
are  the  cockpit  interface  (e.g.,  displays,  controls),  the  aircraft  environ¬ 
ment  (e.g.,  weather  conditions,  visibility,  turbulence)  and  stress,  the 
effects  of  which  cannot  readily  be  segregated.  Thus  in  most  tests,  handling 
qualities  are  really  being  evaluated  in  the  aggregate. 

In  appendix  A,  "Handling  Qualities"  is  defined  as  "those  qualities  or 
characteristics  of  an  aircraft  that  govern  the  ease  and  precision  with  which 
a  pilot  is  able  tc  perform  the  tasks  required  in  support  of  an  aircraft  role." 
The  generally  accepted  meaning  of  "living  Qua! i ties"  is  similar  to  this  defi 
nition  of  "Handling  Qualities,"  so  only  the  latter  term  is  used  in  this 
report  . 
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Figure  1  illustrates  ..it  factors  besides  stability  and  control  that 
influence  handling  qualities.  Here  the  primary  elements  of  the  pilot  control 
loop  are  arranged  to  illustrate  their  relationship  to  the  operation  of  the 
pi  lot- vehicle  combination.  in  addition  to  the  pilot,  the  task,  and  the 
stability  arid  control  characteristics,  factors  shown  as  influencing  closure  of 
the  pilot  control  loops  are  the  cockpit  interface,  the  aircraft  environment, 
and  the  pilot's  stress. 


Figure  1.  L  lenient  s  of  control  loop  that  influence  handling  qualities. 


Mission.-  The  term  "mission"  has  been  used  in  the  United  States  rather 
loosely,  anU  may  actually  have  several  meanings,  depending  on  how  it  is  used. 
"Mission"  has  been  used  to  identify,  in  a  general  sense,  the  purpose  or 
objective  for  which  an  aircraft  is  built.  It  has  also  been  used  to  designate 
a  complete  flight  or  sortie  or  even  an  undefined  part  of  the  flight.  By 
inference,  this  undefined  part  is  usually  the  special  flight  phase  during 
which  the  primary  assignment  is  carried  out. 


To  avoid  this  ambiguity,  the  terms  "role,"  "flight"  or  "sortie,"  "flight 
phase,"  and  "flight  s’jhphase"  are  defined  for  use  in  place  of  "mission"  (or 
mission  element).  The  continued  use  of  "mission"  in  relation  to  handling 
qualities,  however,  makes  it  worthwhile  to  have  an  acceptable  definition. 

One  tl>. at  has  been  suggested  (ref.  4)  is  "the  composite  of  pilot  vehicle  func¬ 
tions  that  must  be  performed  to  fulfill  operational  requirements."  In  the 
present  report,  "mission"  is  a  general  term  used  to  convey  this  concept  of 
"operational  requirements,"  that  is,  the  objectives  or  delineation  of  what  it 
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is  that  the  pi  lot -vehicle  combination  must  be  able  to  accomplish.  As  a  rule, 
the  mission  objectives  (r  qui red  operations)  are  cited  in  more  detail  as  the 
flight  segment  (flight  phase  or  subphase)  of  interest  becomes  more  specific. 

The  distinct  differences  ’  tended  between  "mission"  and  "task"  will  he 
clarified  in  the  following  paragraphs. 

Task.-  The  term  "task"  also  has  various  connotations.  We  are  concerned 
here  only  with  the  pilot's  task,  which  includes  controlling  the  aircraft  as 
well  as  associated  functions,  not  directly  related  to  controlling  the  aircraft, 
such  as;  navigation  and  communications.  A  task  in  the  sense  that  it  is  used  in 
handling  qualities  evaluations  is  defined  as  "the  actual  work  assigned  a  pilot 
to  be  performed  in  completion  of,  or  as  representative  of,  a  designated  flight 
segment."  In  being  representative  of  a  flight  phase,  for  example,  the  impor¬ 
tant  pi lot-vehi cle  functions  required  to  fulfill  the  operational  requirements 
for  that  flight  phase  would  be  represented  in  the  task.  Use  of  "task"  and 
"mission"  differs  then,  in  that  a  task  represents  what  the  pilot  is  actually 
asked  to  do  (as  in  a  simulation  task)  while  a  mission  refers  to  all  opera¬ 
tional  requirements  the  pilot-vehicle  combination  must  be  able  to  accomplish 
if  the  "intended  use"  of  the  aircraft  is  to  be  fulfilled. 

It  is  convenient  to  consider  the  complete  task  to  be  composed  of  (1)  the 
control  task,  and  (2)  auxiliary  tasks.  The  control  task  requires  actuation 
of  the  principal  controls  and  the  selectors  as  required.  The  auxiliary  tasks 
involve  the  pilot  in  actions  other  than  direct  control  of  the  aircraft. 

Flight  phase.-  The  terms  "flight,"  "flight  phase,"  and  "flight  subphase" 
denote  the  flight  profile  of  an  aircraft  and  its  subdivision  into  convenient 
segments.  The  delineation  of  aircraft  role  and  the  division  of  a  complete 
flight  into  discrete  segments  for  more  definitive  examination  is  illustrated 
in  figure  2.  Representative  examples  of  what  is  meant  by  aircraft  role,  com¬ 
plete  flight,  flight  phase,  and  sui  phase  are  given  in  this  figure  as  well  as 
in  appendix  A.  A  subphase  is  defined  as  "that  part  of  a  flight  phase  having 
a  single  objective,  and  a  single  configuration  or  change  in  configuration." 

A  subphase  evaluation  then  would  provide  a  direct  correlation  between  a 
specific  set  of  stability  and  control  parameters  and  pilot  rating. 

In  summary  then,  the  role  of  an  aircraft  defines  its  intended  use  only 
in  a  general  sense.  The  mission  delineates  this  use  in  terms  of  specific 
objectives,  that  is,  the  required  operations  of  the  pi lot-veh i cle  combination. 
The  task  delineates  those  aspects  of  the  mission  that  are  work  assigned  to  the 
p i  lot . 

Methods  of  Determining  Aircraft  Handling  Qualities 

The  relat ion;  h ip  between  stability  and  control  parameters  and  the  degree 
uf  suitability  of  the  airplane  for  the  mis;  ion  may  be  examined  bv: 


' '  e>  -fun!  A'>r’‘<  .;ss  Ti*«*  '  P  *''•< r  '•>  *'  He  {:■(••  f<v  r^<3  <n  1»»*  gnntf*-;  *  qr-‘  .  r>nst»s 


figure  2.-  The  relationship  between  role,  flight  segments,  and  task. 


1.  Theoretical  analysis 

2.  Experimental  performance  measurement 

a.  Pilot  input 

b.  Pilot-vehicle  output 

3.  Pilot  evaluation 

Each  approach  has  an  important  part  in  the  complete  evaluation.  One 
might  ask,  however,  "Why  is  the  pilot  assessment  necessary?”  The  answer  must 
consider  the  two  alternatives,  theoretical  analysis  and  performance  measure¬ 
ment.  At  present,  the  applicability  ot  the  mathematical  analysis  including 
representation  of  the  human  operator  is  restricted  to  the  analysis  of  speci¬ 
fic  simple  tasks.  Since  the  intended  use  (mission)  is  made  up  of  several 
tasks  and  several  modes  of  pilot-vehicle  behavior,  it  is  difficult  first  to 
describe  accurately  all  modes  analytical!;.,  and,  second  to  integrate  the  qual¬ 
ity  in  the  separate  tasks  into  a  measure  of  overall  quality  for  the  intended 
use.  Theoretical  analysis  is  fundamental  to  the  analytical  prediction  of 
handling  qualities,  but  cannot  adequately  treat  the  complex  interactions  that 
are  now  investigated  by  means  of  experimental  pilot  evaluation. 

The  attainment  of  satisfactory  performance  in  fulfilling  a  designated 
mission  is,  of  course,  a  fundamental  reason  for  our  concern  with  handling  qual¬ 
ities.  Why,  men,  cannot  the  experimental  measurement  of  performance  replace 
pilot  evaluation'.’  Why  not  measure  pilot-vehicle  output  performance  in  the 
intenled  use9  Isn't  good  performance  consonant  with  good  quality?  Unfortu¬ 
nately,  the  answer  to  the  latter  question  is  "not  always." 
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A  significant  difficulty  arises  here  in  that,  first,  the  tasks  selected 
for  measuring  performance  nay  not  demand  of  the  pilot  all  that  the  real  mis¬ 
sion  demands,  especially  in  terms  of  distractions,  auxiliary  tasks,  and  pilot 
stress.  In  the  second  place,  pi  lor  performance  must  also  he  measured  and 
interpreted  so  that  the  pilot-vehicle  performance  can  he  evaluated  correctly. 
The  pilot  is  an  adaptive  controller  whose  goal  (when  he  is  so  instructed)  is 
to  achieve  good  performance.  In  a  specific  task,  he  is  capable  of  attain¬ 
ing  essentially  the  same  performance  for  a  wide  range  of  vehicle  characteris¬ 
tics,  at  the  expense  of  significant  reductions  in  his  capacity  to  assume  other 
duties  and  to  plan  subsequent  operations.  Significant:  differences  in  his  task 
performance  may  not  he  measured  when  very  real  differences  in  mission  suit¬ 
ability  do  exist.  The  pilot's  performance  must  be  measured  and  analyzed 
properly  to  show  these  differences,  and  one  must  have  a  clear  appreciation  of 
what  constitutes  objectionable  inputs. 

In  the  third  place,  it  is  difficult,  if  not  impossible,  at  the  present 
time  to  measure  all  important  aspects  of  pilot  performance.  F.ncouraging 
results  have  been  obtained  in  specific  instances  (refs.  5-7)  wherein  good  cor¬ 
relation  has  been  obtained  between  measurements  t  c  the  physical  effort 
exerted  by  the  pilot  (i.e.,  integral  of  pilot  control  displacement,  force, 
etc.)  and  pilot  rating.  In  such  cases,  it  must  be  assumed  that  differences  in 
mental  effort  and  attention  were  not  significant.  Rather  than  attempt  to  pre¬ 
judge  the  influence  of  mental  effort  and  attention,  the  term  workload  is 
defined  to  include  both  mental  and  physical  effort.  The  use  of  modifying 
terms  will  be  necessary  then  when  a  distinction  is  necessary  between  mental 
and  physical  workload. 

The  questions  that  arise  in  using  performance  measurements  may  be 
summarized  as  follows:  (1)  For  what  maneuvers  and  tasks  should  measurements 
be  made  to  insure  fulfillment  of  the  mission  objectives?  (2)  How  do  we  inte¬ 
grate  and  weigh  the  performance  in  several  tasks  to  get  an  overall  measure  of 
quality  if  measurable  differences  do  exist?  (5)  Is  it  necessary  to  measure  or 
evaluate  pilot  workload  for  the  performance  to  be  meaningful?  If  so,  how  are 
these  factors  weighed  with  those  in  (2)?  (4)  What  disturbances  and  distrac¬ 

tions  are  necessary  to  provide  a  realistic  workload  for  the  pilot  while  his 
performance  in  a  specified  task  is  being  measured?  The  difficulties  encoun¬ 
tered  in  answering  each  of  these  questions  provide  some  of  the  reasons  why  con¬ 
tinued  reliance  upon  pilot  evaluation  is  necessary  and  why  much  of  the 
detailed  discussion  is  included  in  this  report.  Although  the  use  of  measured 
task  performance  is  not  discussed  further  in  this  report,  continued  efforts 
to  measure  ard  interpret  both  the  pilot  performance  and  the  pilot-vehicle 
performance  should  be  encouraged.  Such  information  is  important  to  the  under¬ 
standing  of  pilot  adaptation  or  "learning  curves,"  and  to  the  interpretation 
of  pilot  evaluation  data. 

Pile:  evaluation  still  remains  the  only  method  of  assessing  the 
interactions  between  pilot-vehicle  performance  and  total  workload  in  determin¬ 
ing  suitability  of  an  airplane  for  the  mission.  It  provides  a  basic  measure 
of  quality  and  serves  as  a  standard  with  which  pi  lot -airplane  system  fheorv 
may  be  developed,  performance  measurements  may  be  correlated,  and  significant 
airplane  design  parameters  and  characteristics  may  be  determined  and 
cor re  la  ted. 
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Pilot  evaluation  data  generally  consists  of:  (1)  the  pilot  rating,  or 
shorthand  representation  of  the  flying  characteristics  as  they  relate  to  mis¬ 
sion  accomplishment,  and  (2)  the  pilot  comments  that  identify  those 
characteristics  that  interfere  with  the  intended  use. 

A  pilot  rating  is  a  portion  of  the  technical  report  of  the  evaluator,  and 
is  the  overall  summation  of  the  suitability  of  the  vehicle  for  the  specified 
use.  The  pilot  rating  scale  is  then  a  systematic  means  of  denoting  the  qual¬ 
ity  of  the  pi  lot- vehicle  combination  in  the  accomplishment  of  its  intended 
purpose . 


EARLY  RATING  SCALES 


In  early  handling  qualities  research,  each  investigator  tended  to  develop 
a  rating  scale  peculiar  to  the  needs  of  his  specific  program,  or  to  modify  an 
existing  one.  With  experience,  certain  pilot  rating  scales  proved  successful 
and  stimulated  further  interest  in  them.  Two  such  early  scales  are  discussed 
h  -e . 


In  reference  8  the  original  Cooper  Scale  (fig.  3)  proposed  the  basic 
framework  of  boundaries  that  is  still  the  foundation  of  most  pilot  rating 
scales,  including  the  presently  proposed  one.  This  framework  involves  several 
grades  of  quality  pertaining  to  the  intended  use  of  the  vehicle. 
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Important  contributions  of  the  early  scales  were  to  emphasize  pilot  acceptance 
of  handling  qualities  with  respect  to  the  intended  use  by: 

a.  Encouraging  investigators  to  define  adequately  the  program  objectives 
and  the  intended  use  of  the  aircraft. 

b.  Firmly  establishing  within  the  scale  the  acceptable-unacceptable,  and 
satisfactory-unsatisfactory  boundaries . 

c.  Creating  a  logical  basis  that  would  enable  the  pilot  to  express  his 
assessment  accurately  and  consistently. 

The  original  Cooper  Scale  included  too  many'  different  concepts.  The 
introduction  of  stability  augmentation  and  undefined  failure  modes,  of  normal 
and  of  emergency  operation,  and  the  separation  of  the  landing  task  from  the 
primary  mission  led  to  ambiguous  interpretations. 

The  CAL  scale  of  figure  4  was 
developed  primarily  because  the 
Cooper  Scale  was  confusing  to  some  in 
that  it  could  be  interpreted  as  intro¬ 
ducing  an  alternate  mission  concept. 
Separate  boundaries  were  shown  in  the 
Cooper  Scale  for  normal  operation  and 
for  an  undefined  emergency  condition. 
By  removing  this  doubt  of  mission 
completion  in  the  adjective  descrip¬ 
tions  in  the  acceptable  range,  as 
well  as  removing  all  consideration  of 
an  alternate  mission  from  the  scale 
itself,  the  CAL  scale  clarified  this 
situation.  However,  the  very  simple 
descriptions  of  the  CAL  scale  are  not  considered  particularly  helpful  by  many 
pilots  and  the  dual  use  of  "fair"  and  "bad"  was  confusing  to  some. 

Having  now  identified  certain  deficiencies  and  objections  to  two  rating 
scales  and  clarified  some  of  the  terminology,  we  are  in  a  position  to 
construct  a  revised  scale. 


REVISED  PILOT  RATING  SCALE 


Major  Categories 

Category  selection.-  The  Cooper  and  CAL  Rating  Scales  and  the  revised 
scale  proposed  in  references  1  and  2  have  the  same  basic  structure.  The  major 
categories  are  identified  as  "satisfactory"  and  "unsatisfactory,"  or  "accept¬ 
able,"  "unacceptable  "  and  "uncontrollable."  Such  terms  relate  only  to  the 
individual  pilot's  assessment  of  quality  relative  to  the  jnt  ided  use  of  the 
aircraft.  The  intention  has  been  to  encourage  the  pilot  t  ,ake  the  important 
decisions  identified  with  these  terms  .  These  categories  .  svstemat  i  -'a!  1'- 
arranged  so  that  the  pilot  can  choose  between  two  alu-vr  that  lead  to 

the  p r 0 p  e r  c  a t ego  r  y . 
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Figure  4.-  CAL  Rating  Scale. 
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This  structure  is  illustrated  by 
the  flow  chart  in  figure  5,  which 
enables  one  to  trace  the  series  of 
dichotomous  decisions  the  pi  lot:  makes 
in  arriving  at  the  final  rating.  As 
a  rule,  the  first  decision  is  fairly 
obvious.  Are  the  handling  qualities 
controllable  or  uncontrollable?  To 
determine  whether  this  decision 
applies  throughout  the  task  or  flight 
phase  and  in  context  of  the  defined 
role  may  not  be  so  obvious. 

If  the  vehicle  is  uncontrollable, 
it  is  rated  10.  If  it  is  control¬ 
lable,  the  second  decision  is  whether  it  is  acceptable  or  unacceptable.  If 
unacceptable,  the  rating  7,  8,  or  9  will  be  selected  (rating  10  has  been 
excluded  by  the  "controllable"  answer  to  the  first  decision).  If  it  is  accept¬ 
able,  the  third  decision  is  whether  it  is  satisfactory  or  unsatisfactory.  If 
unsatisfactory,  the  rating  4,  5,  or  6  will  be  selected;  if  satisfactory,  the 
rating  1 ,  2 ,  or  3  will  be  selected. 

Category  definitions.-  The  dichotomous  decisions  outlined  in  the  previous 
paragraph  logically  lead  to  four  categories  of  quality,  some  of  which  have 
been  difficult  to  describe  accurately  by  a  single  adjective.  There  are  objec¬ 
tions  to  the  designation  "unsatisfactory  but  acceptable"  and  "un accept  able  but 
controllable,"  but.  no  simple  adjective  descriptions  for  these  categories  have 
been  found  that  will  satisfy  everyone.  In  spite  of  strong  emphasis  as  to  the 
need  to  establish  clearly  that  the  terms  are  related  to  the  pilot’s  oum 
assessment  of  acceptability  and  not  to  any  existing  standards,  specifications, 
or  other  Acceptance  Criteria,  objection  to  "acceptable"  and  "unacceptable"  in 
the  scale  remains. 

Regardless  of  the  terms  used  in  a  revised  rating  scale,  it  is  necessary 
to  clarify  the  intended  meaning  of  satisfactory,  unsatisfactory,  acceptable, 
tolerable,  unacceptable,  controllable,  and  uncontrollable.  Let  us  examine, 
first,  what  is  meant  by  "controllable,"  even  though  it  has  caused  less  diffi¬ 
culty  than  the  other  terms.  To  control  is  to  exercise  direction  of,  to  com¬ 
mand,  or  to  regulate.  The  determination  as  to  whether  the  airplane  is 
controllable  must  be  made  within  the  framework  of  the  defined  mission  or 
intended  use.  An  example  of  the  considerations  of  this  decision  would  be  the 
evaluation  of  fighter  handling  qualities  during  which  the  evaluation  pilot 
encounters  a  situation  in  which  he  can  maintain  control  only  with  his  complete 
and  undivided  attention.  The  vehicle  is  "controllable"  in  this  situation  in 
the  sense  that  the  pilot  can  maintain  control  only  by  restricting  the  tasks 
and  maneuvers  he  is  called  upon  to  perform  and  by  giving  the  configuration  his 
undivided  attention.  However,  for  him  to  answer,  "Yes,  it  is  controllable  in 
the  flight  phase  (or  task),"  he  must  be  able  to  retain  control  in  all  mission 
oriented  tasks  and  other  required  operations  without  sacrificing  effort  and 
attention  to  his  overall  duties. 


i  (  ontrolinbi«? 


Uncontrollable 


Acceptor* 


u^ric ceptQbie  ! 


V  .  Satisfactory  I  or  j  unsoti?.foctory  I 

~r4- ^  : 


1 


i - 1 — i  r 

4  5  6  7 


8  9  10 

Figure  S.-  Sequential  p:  1 ot - ra t ing  decisions. 
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Consider  now  the  meaning  of  acceptable,  the  usage  of  which  has  been  the 
subject  of  some  controversy,  and  the  alternate  suggested  term  "tolerable." 

The  dictionary  shows  "acceptable"  to  mean  that  a  thing  offered  is  received 
with  a  consenting  mind;  "unacceptable"  to  mean  that  it  is  refused  or  rejected. 
Acceptable  means  that  the  flight  phase  (or  task)  can  be  accomplished;  it 
means  that  the  evaluation  pilot  would  agree  to  use  it  for  the  designated  role; 
that  such  deficiencies  as  may  exist  can  be  endured  or  tolerated.  Use  of  the 
term  "acceptable"  does  not  say  how  good  it  is,  but  it  does  say  the  pilot  con¬ 
siders  it  good  enough  for  the  intended  use.  With  these  characteristics,  the 
flight  phase  (or  task)  can  be  accomplished  with  adequate  precision.  The  task, 
for  example,  may  be  accomplished  with  considerable  effort  and  concentration 
on  the  part  of  the  pilot,  but  the  level  of  workload  required  to  achieve  this 
performance  is  tolerable  and  not  unreasonable  in  context  with  the  intended 
use.  By  the  same  token,  "unacceptable"  does  not  necessarily  mean  that  the 
designated  flight  phase  (or  task)  cannot  be  accomplished;  it  does  mean  that 
the  necessary  performance  cannot  be  achieved  or  that  the  effort,  concentration, 
and  workload  required  are  of  such  magnitude  that  the  evaluation  pilot  rejects 
the  aircraft  for  this  phase  of  its  intended  use. 

Consider  now  a  definition  of  "satisfactory."  The  dictionary  defines  this 
as  adequate  for  the  purpose,  of  a  kind  to  meet  all  requirements  or  expecta¬ 
tions.  A  pilot's  definition  of  satisfactory  might  be  that  it  isn't  neces¬ 
sarily  perfect,  or  even  good,  but  it  is  good  enough  that  he  wouldn’t  ask  that 
it  be  changed.  It  meets  a  standard;  it  has  sufficient  goodness;  it's  of  a 
kind  to  meet  all  pilot  demands  for  the  intended  use. 

Unsatisfactory  implies  that  there  is  insufficient  goodness  to  meet  all 
pilot  demands;  that  it  has  deficiencies  and  objectionable  characteristics 
which  he  feels  should  be  corrected.  Unsatisfactory  includes  all  that  is  not 
satisfactory,  just  as  acceptable  includes  all  that  is  not  specifically  refused 
as  unacceptable.  A  specific  category  is  then  that  which  is  "unsatisfactory 
but  acceptable."  This  category  has  previously  been  referred  to  by  the  shorter 
term  "acceptable"  but  could  also  be  referred  to  as  "tolerable"  in  the  sense, 
"capable  of  being  borne,  supportable,  bearable."  As  a  result  of  the  possibil¬ 
ity  of  the  misinterpretation  in  the  use  of  "acceptable"  it  has  been  found 
preferable  to  use  "tolerable"  rather  than  "acceptable"  in  the  rating  scale. 

Thus,  the  quality  is  either: 

a.  Satisfactory  -  good  enough  without  improvement,  and,  therefore,  of 
the  best  category,  or 

b.  Unsatisfactory  but  tolerable  -  just  good  enough,  adequate  for  the 
purpose  but  improvement  desirable,  and,  therefore,  of  the  next  best  category, 
or 

c.  Unacceptable  to  the  pilot  -  not  suitable  for  the  purpose  but  still 
controllable,  and  in  the  third  category,  or 

d.  Uncontrollable  unacceptable  for  the  purpose  and  of  the  poorest 
quality  arid  in  the  fourth  category. 
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Inasmuch  as  only  four  categories  of  quality  are  needed  to  identify  and 
describe  handling  qualities,  it  is  possible  to  simplify  the  dichotomous 
decision  process  illustrated  in  figure  5  and  eliminate  "acceptability"  terms. 

By  considering  the  following  three  decisions  the  pilot  will  arrive  at  one 
of  the  four  categories  pi'eviously  discussed: 

1.  Is  the  vehicle  controllable? 

2.  Is  adequate  performance  attainable  with  a  tolerable  workload? 

3.  Is  the  vehicle  satisfactory  without  improvement? 


Being  able  to  designate  a 
category  by  a  single  adjective  was 
considered  less  important  than  that 
the  definition  of  each  category  be 
precise.  The  rating  scale  now  pro¬ 
posed  is  therefore  based  on  the  cate¬ 
gory  definitions  provided  in  figure  6. 
These  are  reasonably  concise  and 
emphasize  the  pilot's  decisions 
involved  in  category  selection. 

Fundamental  to  this  selection  is 
the  consideration  of  whether  adequate 
performance  is  attainable  in  the 
selected  task  or  required  operation 
(mission)  and  whether  or  not  defici¬ 
encies  are  present  which  require 
pilot  compensation. 

Assessment  of  task  performance 
and  pilot  workload  then  enables  the 
pilot  to  decide  whether  the  handling 
qualities  are  good  enough  without 
improvement,  have  deficiencies  for 
which  improvement  is  desired,  or  have 
deficiencies  for  which  improvement  is 
essential . 


Individual  Ratings 

The  complete  revised  rating  scale1  in  figure  7  includes  further 
subdivisions  of  quality  within  three  of  the  four  major  categories  with  appro¬ 
priate  descriptions  for  each  numerical  rating  to  define  quality  differences. 
These  allow  the  pilot  a  sufficient  range  of  handling  quality  descriptions  for 
most  situations.  Repetition  of  descriptive  terms  from  the  category  defini¬ 
tions  has  all  but  been  eliminated  in  the  individual  rating  descriptors. 

^Smal 1  and  large  copies  of  figure  7  are  available.  Send  requests  to 
Technical  Information  Division,  Ames  Research  Center,  Moffett  Field,  Calif., 

94035 . 
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ADEQUACY  FOR  SELtOltD  TAb*  OR 
REQUIRED  OPERATION* 
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Figure  7.-  Handling  Qualities  Rating  Scale. 


Considered  desirable  in  simplifying  the  scale  for  the  experienced  user,  this 
arrangement,  however,  makes  it  mandatory  that  any  user  understand  and  utilize 
the  category  definitions  and  make  the  decisions  outlined  on  the  left.  The 
important  "boundary"  decisions  between  pilot  ratings  of  3  and  4,  6  and  7,  or 
9  and  10  cannot  be  made  by  reference  to  the  individual  rating  descriptors 
alone.  It  is  emphasized  that  these  descriptions  supplement  the  sequential 
decisions  that  lead  the  evaluation  pilot  to  the  particular  category  within 
which  the  descriptions  of  the  individual  ratings  are  given.  It  was  considered 
fundamental  to  a  good,  easily  applied  scale  that  the  descriptions  be  both 
brief  and  general.  Key  words  and  phrases  were  sought  that  would  easily  he 
understood  and  yet  sufficiently  definitive  so  that  each  rating  would  be 
clearly  separated  from  every  other  rating.  The  following  paragraphs  discuss 
and  explain  some  of  the  factors  related  to  the  rating  scale 


Performance  and  workload.-  Consideration  was  given  to  describing  quality 
in  terms  of  both  performance  (precision  of  aircraft  control J  and  workload 
(effort  and  attention)7  for  each  numerical  rating.  Upon  closer  examination, 

^When  applied  to  handling  qualities,  the  term  "performance"  alone  is 
intended  to  mean  the  precision  of  aircraft  control  attained  by  the  pilot,  that 
is,  the  pilot-vehicle  performance.  Ihe  term  "workload"  is  intended  to  convey 
the  amount  of  effort  and  attention,  both  physical  and  mental,  that  the  pilot 
must  provide  to  attain  a  given  level  of  performance.  "Pilot  performance"  may 
be  used  to  describe  the  measure  of  physi val  workload  (effort  and  attention) 
used  in  performing  a  task. 
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however,  it  was  concluded  that  these  factors  were  so  interdependent  and 
subject  to  tradeoff  by  a  pilot  that  it  would  be  impossible  to  define  non- 
conflicting  individual  ratings.  Precision  of  control  could  not  be  defined 
independently  of  the  amount  of  effort  and  attention  required  of  the  pilot  for 
any  task . 

The  best  category  is  defined  as  desired  or  clearly  adequate  performance 
in  association  with  a  satisfactory  level  of  pilot  workload.  The  second  best 
category  is  defined  by  achievement  of  adequate  performance  although  the  pilot 
is  required  to  compensate  for  deficiencies  by  increasing  his  workload.  It  is 
tolerable  to  him  but  he  desires  improvement.  The  third  category  involves 
major  deficiencies  because  the  pilot  finds  it  impossible  to  achieve  adequate 
performance  even  though  his  attempt  to  compensate  increases  the  total  workload 
to  the  maximum  tolerable  in  context  of  the  task  or  mission.  Uncontrollable 
obviates  further  concern  with  performance,  but  as  in  all  previous  categories, 
must  also  be  considered  in  the  context  of  the  selected  task  or  required 
operation  (mission) . 

Compensation . -  "Pilot  compensation"  as  used  in  the  scale  is  intended  to 
indicate  that  the  pilot  must  increase  his  workload  to  improve  aircraft  perfor¬ 
mance.  It  relates  the  pilot's  difficulty  in  completing  a  task  with  the 
precision  required  for  that  task.  Stated  another  way,  it  is  the  measure  of 
additional  pilot  effort  and  attention  required  to  maintain  a  given  level  of 
performance  in  the  face  of  less  favorable  or  deficient  characteristics.  Hie 
total  workload  is  then  comprised  of  the  workload  due  to  compensation  for  air¬ 
craft  deficiencies  plus  the  workload  due  to  the  task. 

Referring  to  figure  6,  we  see  that  it  is  really  only  necessary  to  define 
that  which  constitutes  adequate  performance  when  answering  the  second  question. 
The  precision  of  control  required  for  any  task  is  most  easily  defined  in  terms 
of  the  end  result  obtained.  lor  example,  the  approach  performance  can  be  spec¬ 
ified  in  terms  of  the  threshold  "gate,"  and  landing  performance  is  measured  in 
terms  of  vertical  velocity  at  touchdown  and  dispersion  about  the  intended 
touchdown  point.  The  necessity  for  holding  airspeed,  altitude,  flight  path, 
or  other  parameters  within  specified  limits  throughout  each  flight  phase  or 
subphase  may  also  be  specified,  but  the  cause  and  effect  of  occasionally 
exceeding  the  limits  may  require  pilot  interpretation.  The  pilot  must  balance 
both  aspects  of  performance  -  maintaining  precision  of  control  and  the  end 
result  -  against  his  own  effort  and  attention  in  arriving  at  a  rating. 

The  category  definitions  of  figure  6  recognize  the  interaction  between 
performance  achieved  and  the  pilot  workload  required  to  compensate  for  defi¬ 
cient  characteristics.  In  the  best  category,  no  significant  compensation,  in 
terms  of  added  workload,  is  required  by  the  pilot  to  achieve  adequate  perfor¬ 
mance.  In  fact,  adequate  performance  is  clearly  achievable  and  it  may  just 
as  easily  be  possible  to  attain  some  higher  level  of  performance,  such  as 
might  be  designated  broadly  as  desired  performance.  From  the  definition,  it 
may  be  deduced  that  desired,  or  at  least  clearly  adequate ,  precision  of  con¬ 
trol  can  be  obtained  with  relative  ease  or  a  low  level  of  effort  and  atten¬ 
tion.  With  deficiencies  characterized  by  the  second  category,  considerably 
more  pilot  effort  is  required  and  desired  performance  is  not  necessarily 
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obtained.  Now,  however,  a  lower  level  of  handling  performance,  but  one 
adequate  for  the  purpose,  can  be  achieved  as  long  as  the  pilot  will  increase 
his  workload,  even  to  the  maximum  tolerable  for  the  selected  task.  If  major 
deficiencies  exist,  adequate  precision  of  control  would  require  an  excessive 
level  of  pilot  workload  and  even  the  maximum  tolerable  pilot  workload  will  not 
enable  achievement  of  the  level  of  performance  that  is  considered  adequate 
for  the  selected  task.  In  this  category,  the  additional  deficiencies  that 
increase  demands  on  the  pilot  can  only  be  expected  to  result  in  inadequate 
performance,  and  the  primary  question  becomes  one  of  controllability.  The 
manner  in  which  precision  of  control  may  be  traded  off  against  pilot  effort 
and  attention  is  complex  and  defies  explicit  definition  for  all  flight  phases 
or  specified  tasks.  However,  such  compensation  is  available  to  the  pilot  and 
is  probably  involved  in  the  evaluation  of  handling  qualities. 

Failures  or  emergency  operation.-  The  revised  scale  contains  no  reference 
to  failure  considerations  or  emergency  operation.  In  effect,  this  means  that 
the  pilot  need  not  always  concern  himself  during  an  evaluation  with  the  prob¬ 
ability  of  a  failure,  nor  with  the  length  of  time  he  might,  be  faced  with 
deficient  handling  qualities.  The  time  (duration)  during  which  particular 
characteristics  must  be  coped  with  is  inherently  defined  by  the  role  and  the 
flight  phase  or  task  being  evaluated.  For  operation  based  on  a  normal  air¬ 
craft  state,  the  duration  can  be  based  explicitly  on  the  aircraft  rcle.  If  a 
failure  occurs,  however,  it  may  be  necessary  to  consider  alternatives  like 
aborting  the  flight  or  changing  the  flight  plan  to  permit  retreat  to  a  more 
favorable  flight  condition;  in  either  case,  the  length  of  time  that  the  criti¬ 
cal  flight  condition  must  be  tolerated  might  be  shortened.  Unless  such 
alternatives  are  spelled  out  in  the  task  definition,  however,  the  pilot  must 
always  treat  a  failure  state  as  having  to  be  coped  with  for  the  duration  of 
the  task  or  flight  phase,  depending  on  which  is  being  rated.  Normal  and 
emergency  states  then  will  require  separate  evaluations. 

There  are  environmental  conditions  as  well  as  minor  failures  which  can 
limit  aircraft  operation  without  being  identified  as  an  emergency.  Fxcessivc 
turbulence  or  crosswind  could  preclude  attempting  a  landing  and  thus  require 
that  the  flight  plan  be  altered  accordingly.  Whenever  iimitations  to  the 
operation  are  indicated  or  accepted  in  lieu  of  specifying  a  critical  task  this 
fact  should  be  clearly  noted. 

Pilot  skill.-  The  pilot  effort  and  attention  required  for  a  given  task  or 
flight  pTiase  will,  of  course,  depend  somewhat  upon  individual  pilot  skill  and 
state  of  training.  In  the  early  versions  of  the  revised  scale,  the  term 
'‘skill"  was  used  in  the  8  and  9  rating  descriptions  in  consideration  of  the 
role  of  pilot  skill  in  the  dt gree  of  controllability  which  could  be  achieved. 
Further  consideration,  however,  has  led  in  the  8  and  9  rating  descriptions 
to  the  substitution  of  phrases  reflecting  pilot  workload  required  to  even  con¬ 
tinue  the  task  and  the  relative  difficulty  in  maintaining  control,  in  order  to 
retain  a  consistency  and  compatibility  with  the  rest  of  the  scale.  In  this 
manner,  pilot  skill  becomes  a  consideration  of  the  total  program  for  which  the 
evaluation  is  being  made. 
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It  is  the  opinion  of  the  authors  that  as  a  general  rule  each  pilot  should 
judge  the  suitability  of  any  set  of  airplane  characteristics  in  terms  of  his 
own  skill  and  training,  and  in  terms  of  the  required  operations  and  circum¬ 
stances  as  defined  in  the  experiment.  The  effects  of  differing  skills  should 
be  determined  from  the  results  obtained  from  evaluation  pilots  of  different, 
but  representative,  levels  of  experience  and  training.  Exceptions  to  this 
general  rule  have  occurred,  however,  when  the  research  or  development  test 
pilot  is  asked  to  evaluate  handling  qualities  with  respect  to  his  understand¬ 
ing  of  the  lowest  degree  of  skill  and  training  existent  in  a  group  of 
operational  pilots. 

Operating  margins  and  safety.-  The  question  of  safety  cannot  be  separated 
from  handling  qualities  because  the  precision  with  which  certain  parameters 
are  controllable  determines  whether  available  safety  margins  are  apt  to  be 
exceeded.  The  margins  provided,  as  well  as  the  consequence  of  exceeding  them, 
will  certainly  influence  the  effort  and  attention  which  a  pilot  applies.  Oper¬ 
ating  margins  with  respect  to  such  parameters  as  airspeed,  angle  of  attack, 

Mach  number,  and  altitude  then  represent  the  constraints  that  define  required 
precision  of  control,  and  thereby  influence  the  rating  selection.  Operating 
margins  and  safety  should  be  recognized  as  constraints  upon  handling  qualities 
that  are  inextricably  related  to  performance  and  workload. 


Considerations  Associated  With  the  Structure  of  the  Revised  Scale 

Why  a  10-point  scale?-  In  discussing  the  revised  scale,  one  question 
that  might  be  anticipated  concerns  the  number  of  individual  ratings  the  scale 
defines.  Most  simply,  the  number  is  related  to  the  four  categories  already 
selected.  Separating  each  of  the  upper  three  categories  into  three  subratings 
appears  to  provide  an  adequate  spread  for  pilot  use.  Additional  ratings  in 
the  fourth  category  (uncontrollable)  would  not  appear  to  be  of  general  value. 

A  change  in  the  number  of  individual  ratings  for  each  category  was  not  deemed 
necessary  or  desirable  when  considered  in  light  of  the  large  amount  of  exper¬ 
ience  with  the  previous  scales. 

Identifying  the  revised  rating.-  The  oft -proclaimed  criticism  that  the 
scale  should  start  with  10  and  progress  to  1  instead  of  from  1  to  10  may  be 
valid,  but  there  are  also  examples  that  support  the  1  to  10  logic.  We  are 
reluctant  at  this  point  to  suggest  a  change  simply  because  of  the  widespread 
use  of  3-1/2  and  6-1/2  boundaries.  To  now  reverse  these  would  likely 
introduce  considerable  confusion  and  would  not  necessarily  have  long-range 
benefits . 

In  proposing  a  revised  scale,  it  was  recognized  that  some  confusion  might 
result  from  continued  use  of  the  same  numerical  scale  that  has  been  identified 
with  both  the  Cooper  and  CAL  Scales.  in  references  1  arid  2,  the  authors  pro¬ 
posed  a  modified  identification  system  using  letters  A  and  U  in  conjunction 
with  the  numbers  1  -  9.  Subsequent  experience  and  comment  indicated  this  to 
be  unnecessary,  as  long  as  the  rating  numerals  are  associated  explicitly  with 
the  particular  scale  used. 
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Linearity. -  This  is  a  desirable  characteristic  of  any  scale.  A 
temperature  scale  is  linear  with  heat  added  for  a  material  with  constant  spe¬ 
cific  heat,  in  that  the  temperature  rise  per  unit  quantity  of  heat  added  is 
the  same  throughout  the  scale.  Temperature  is  a  normal  and  useful  scale  asso¬ 
ciated  with  comfort.  Even  though  temperature  may  be  quite  linear  with  heat 
added,  comfort  is  not  linear  with  temperature.  With  what  should  the  pilot  rat 
ing  scale  be  linear?  Since  it  is  purported  to  measure  quality,  it  should  then 
be  linear  with  the  added  quality  of  the  pi lot-vehi c le  combination  in  that  the 
change  in  pilot  rating  per  unit  quality  addition  should  be  the  same  throughout 
the  rating  scale.  The  rating  scale  may  possibly  have  this  characteristic,  but 
to  demonstrate  that  the  scale  is  indeed  linear  would  require  an  independent 
measure  of  quality  that  does  not  presently  exist.  Since  the  basic  merit  of 
the  scale  is  not  significantly  affected  by  the  lack  of  demonstrated  linearity, 
this  factor  has  not  been  considered  further.  McDonnell  in  reference  4 
describes  a  study  that  establishes  a  correlation  between  the  Cooper  Scale, 
the  revised  scale  of  references  1  and  2,  and  a  linear  scale. 

Ordinal  versus  interval  scale.-  An  interval  scale  is  desirable,  but  the 
proposed  pilot  rating  scale  cannot  be  shown  to  be  an  interval  scale.  The 
authors  have  accepted  it  as  being  ordinal.  It  is,  however,  primarily  an  abso¬ 
lute  scale  rather  than  a  relative  one.  The  pilot  rating  is  given  for  a  con¬ 
figuration  in  the  context  of  its  acceptability  to  the  pilot  for  the  specified 
flight  phase  (or  task)  and  not  in  terms  of  its  goodness  with  respect  to  a  con¬ 
figuration  already  evaluated.  Fortunately,  the  concentration  and  effort 
required  in  performing  each  evaluation  tends  to  suppress  in  the  pilot's  mem¬ 
ory  the  characteristics  of  preceding  configurations,  enabling  him  to  consider 
objectively  each  configuration  on  its  own  merits  for  the  required  operations 
without  continually  making  paired  comparisons.  Pilots  are  reluctant  to  rate 
something  as  excellent  or  optimum  for  fear  that  a  subsequent  configuration 
will  be  better  than  anything  they  considered  possible. 

Words  versus  numbers.-  The  basic  structure  of  the  rating  scale  is 
completely  dependent  on  words  and  their  explicit  definitions.  The  numeral 
associated  with  the  (.‘Valuator's  final  decision  is  an  expedient,  a  shorthand 
symbol.  One  risk  associated  with  a  numerical  scale  is  that  engineers  will 
attempt  to  treat  the  pilot  rating  data  with  mathematical  operations  that  are 
rigorously  applicable  only  to  a  linear  interval  scale.  Although  some  insight 
is  sometimes  gained,  analysis  of  specific  pilot  rating  data  should  not  be 
totally  dependent  on  such  mathematical  operations. 

Differing  standards  of  acceptance  for  the  same  mission.-  One  difficulty 
that  has  arisen  in  the  use  of  pilot  rating  data  can  be- T fTus t rated  hv  the  fol¬ 
lowing  example.  In  an  evaluation  program  for  the  landing  approach  flight 
phase  of  a  commercial  air  transport  the  role  and  mission  were  careful Iv 
defined,  the  program  was  run,  and  the  results  were  reported.  One  of  the  eval¬ 
uation  pilots,  an  airline  pilot,  subsequent lv  remarked  that  his  airline  would 
not  accept  any  airplane  with  worse  than  a  41/2  rating  for  the  landing 
approach.  However,  using  the  generally  accepted  interpretation  of  the  rating 
scale,  this  pilot  had  said,  as  an  evaluation  pilot,  that  aircraft  with  wors“ 
characteristics  which  he  rated  at  5  and  t>  could  still  be  considered  acceptable 
for  the  intended  use.  In  the  first  case  he  was  reflecting  the  viewpoint  of 


airline  management  and  not  speaking  from  the  pilot's  standpoint.  It  must  be 
recognized  that  the  final  determination  of  overall  acceptability  of  an  air¬ 
craft  for  a  given  role  may  be  influenced  by  factors  other  than  just  the 
piloting  rating  of  the  handling  qualities. 

In  deciding  what  to  buy,  any  customer  considers  what  he  will  get  in  terms 
of  how  much  it  will  cost.  It  is  easy  to  envision  similar  decisions  being  made 
to  buy  only  that  which  is  above  the  3-1/2  boundary.  And  similarly,  one  can 
envision  a  reluctant  decision  to  buy  as  low  as  a  6  rating,  or  to  even  accept 
a  7  or  8  rating  (if  the  rating  applied  to  a  flight  condition  or  aircraft  fail¬ 
ure  state  of  low  prcbabi lity)  ,  but  only  if  all  other  possibilities  for  pur¬ 
chase  of  a  better  aircraft  had  been  excluded.  The  basic  pilot  rating  data  on 
which  these  decisions  are  based,  however,  must  be  strictly  mission-oriented  if 
the  subsequent  quality  versus  cost  decisions  are  to  be  meaningful.  That  is  to 
say,  the  role  must  be  understood  and  each  flight  phase  adequately  defined  by 
the  piloting  task(s)  provided. 

EXPERIMENTAL  USE  OF  PILOT  RATING  OF  HANDLING  QUALITIES 


In  previous  usage  of  pilot  rating  scales,  too  little  attention  has  been 
given  to  defining  just  what  the  pilot  was  rating  and  how  the  data  were  to  be 
used.  In  defining  the  experiment  and  in  reporting  the  results,  it  became 
apparent  that  (1)  the  term  "handling  qualities"  requires  a  tighter,  less  ambig¬ 
uous  definition,  (2)  an  accounting  is  required  for  certain  factors  in  addition 
to  stability  and  control,  (3)  a  clearer  understanding  is  needed  of  the  differ¬ 
ence  between  mission  and  task  definition,  (4)  certain  considerations  relative 
to  the  pilots  assessment  criteria  must  be  understood,  and  (5)  a  simulation 
situation  requires  special  definition  and  consideration  of  the  use  of  the 
results  and  of  the  need  for  pilot  extrapolation. 

In  the  following  paragraphs  attention  is  devoted  to  certain  aspects  of 
the  design  and  execution  of  handling  qualities  experiments  that  are  considered 
particularly  important  to  the  production  of  good  data  and  the  further 
clarification  of  the  aforementioned  problems. 


Program  Development 

Fundamental  to  any  handling  qualities  program  is  a  cl^ar  definition  of 
the  primary  objectives  of  the  program  and  of  the  role  and  mission  of  the  air¬ 
craft.  Next  is  the  designation  of  the  tasks  to  be  used  in  the  course  of  the 
evaluation,  and  what  the  rating  applies  to. 

Program  objective  -  Generally,  it  is  expected  that  the  use  of  the  data 
will  conform  reasonably  well  with  the  objectives  outlined  for  the  program. 

The  primary  objectives  are  related  to  either  research,  development,  or  accep¬ 
tance  but  there  may  also  tie  a  number  of  special  considerations,  (These 
include  whethei  the  program  is  exploratory  or  expected  to  be  highly  definitive, 
etc.)  it  is  essential  to  define  whether  stability  and  control  characteristics 
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or  cockpit  interface  elements  are  the  primary  variables  of  investigation  arul 
whether  special  conditions  such  as  environmental  disturbances,  dynamic  or 
static  failures,  or  stresses  are  to  be  included  in  the  task,  or  considered  in 
the  rating.  Of  particular  importance  will  be  the  selection  of  evaluation 
pilots  and  their  indoctrination  with  respect  to  the  program.  The  particular 
background,  training,  experience,  and  point  of  view  of  the  evaluation  pilots 
selected  for  ths  program  may  be  determined  by  the  program  objective  as  well  as 
by  the  intended  use  of  the  resultant  data. 

Mission  description.-  The  explicit  description  of  the  mission  by 
delineation  of  the  'required  operations"  is  probably  the  most  important  con¬ 
tribute!  to  the  objectivity  of  the  pilot  evaluation  data,  ihe  role  must  be 
carefully  analysed  and  a  clear  description  and  understanding  reached  between 
the  engineer  and  the  evaluation  pilot  as  to  their  interpretation  of  the 
required  operations.  This  description  must  include: 

a.  What  the  pilot  is  required  to  accomplish  with  the  aircraft,  and 

b.  The  conditions  or  circumstances  under  which  the  mission  is  to  be 
conducted . 

Because  of  their  importance  in  specifying  "what  the  pilot  rating  applies 
to"  and  in  providing  definitive  guidance  for  the  pilot  in  his  evaluation,  the 
"required  operations"  may  often  have  to  be  given  in  considerable  detail.  !n 
its  simplest  form,  this  consists  of  designating  the  flight  phase  or  subphase(s) 
of  interest,  and  including  such  variations  as  are  considered  critical  and 
representative  of  actual  operations. 

As  an  example  of  (b) ,  the  conditions  or  circumstances  might  include 
instrument  or  visual  flight,  type  of  displays  or  controls  in  the  cockpit,  or 
other  input  information  to  assist  the  pilot  in  accomplishing  the  mission,  etc. 
The  environment  in  which  the  mission  is  to  be  accomplished  must  also  be 
defined  and  considered  in  the  evaluation,  and  could  include,  for  example,  the 
presence  or  absence  of  turbulence,  day  versus  night,  the  frequency  with  which 
the  mission  has  to  repeated,  the  preparedness  of  the  pilot  for  the  mission, 
and  the  p. lot's  level  of  proficiency. 

As  noted  under  the  discussion  of  the  revised  scale,  it  was  recommended 
that  steady-state  failure  considerations,  with  the  attendant  questions  as  to 
probability  of  occurrence,  should  be  removed  from  the  scale.  this  means  that 
separate  ratings  would  be  obtained  for  each  failure  mode.  Steady-state  fail¬ 
ure  modes  would  then  norm  1 1 y  be  considered  as  exist  jig  throughout  the  evalu¬ 
ation  task  or  flight  phase,  unless,  of  course,  a  specific  evaluation  program 
s  designed  to  consider  them  otherwise. 

in  evaluating  the  impact  of  transient  disturbances  caused  by  either 
environmental  conditions  or  system  failures,  tne  pilot  is  still  faced  with  the 
basic  decision  of  what  tradeoff  he  will  accept  between  performance  and  work¬ 
load  and  h ow  long  he  must  cope  with  the  condition. 

High  probability  occurrences,  such  as  iow  to  moderate  turbulence,  wind 
shear,  and  cross  wind,  can  be  rated  separately,  as  if  occurring  throughout  the 
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task  or  flight  phase,  but  should  be  included  in  a  composite  rating  or  a  flight 
phase  rating  to  convey  the  realistic  situation.  It  is  preferable  to  evaluate 
separately  such  low  probabi  ty  occurrences  as  severe  turbulence  so  that  the 
pilot  is  not  asked  to  weigh  probability  of  occurrence  and  generalize  a  compos¬ 
ite  rating  to  the  point,  that  it  loses  value.  For  example,  a  pilot  may  prefer 
to  discuss  a  short  duration  occurrence  of  degraded  handling  qualities  in  the 
pilot  comment  data  rather  than  to  make  a  radical  change  in  the  task  rating  or 
flight  phase  rating. 

Such  questions  of  probability  of  occurrence  and  levels  of  disturbances 
must  be  resolved  as  part  of  the  mission  description  in  the  design  of  the  exper¬ 
iment,  with  special  attention  often  being  required  with  respect  to  pilot 
orientation  and  the  reporting  of  results. 

What  the  Rating  Applies  to 

Task  and  flight  phase  terminology.-  For  a  pilot  to  evaluate  the  handling 
qualities  of  an  aircraft,  he  must  maneuver  it  and  otherwise  use  it  for  its 
intended  purpose.  This  purpose  is  given  in  a  general  sense  by  the  role  but 
more  specifically  by  the  designation  of  the  required  operations  or  mission. 
Next,  it  must  be  decided  whether  the  handling  qualities  are  being  evaluated 
for  a  complete  flight,  a  single  flight  phase,  a  subphase,  or  a  specifically 
defined  task.  In  any  event,  a  task  or  series  of  tasks  must  be  specified  that 
will  provide  a  suitable  basis  for  pilot  evaluation.  The  extent  to  which  the 
evaluation  task  selected  represents  all  aspects  of  the  aircraft  mission  will 
depend  upon  several  factors,  the  more  important  of  which  are  program  objective, 
simulation  capability,  and  certain  pilot  considerations.  From  the  definitions 
provided,  it  will  be  possible  to  establish  conventional  procedures  for  refer¬ 
ring  to  task  ratings  in  contrast  to  flight,  flight  phase,  or  subphase  ratings. 

General  considerations.-  To  provide  an  overall  pilot  rating  for  a 
complete  flight  or  aircraft  role  would  likely  involve  so  many  situations  that 
little  of  the  information  would  relate  to  specific  stability  and  control  char¬ 
acteristics.  For  such  a  use,  an  overall  rating  would  be  of  ittle  value 
unless  reference  were  made  to  the  detailed  comments  provided  by  the  pilot  for 
each  subphase.  A  flight  subphase,  on  the  other  hand,  is  devoted  to  a  single 
objective  and  has  a  single  configuration  or  change  in  configuration  There¬ 
fore  evaluation  of  a  flight  subphase  would  enable  more  direct  correlation  to 
be  made  between  pilot  rating  and  specific  stability  and  control 
character! st ics . 

Should  significant  changes  then  occur  in  either  the  aircraft  state,  the 
cockpit  interface  or  the  aircraft  environment  during  a  given  flight  phase,  it 
might  be  desirable  to  confine  the  pilot  rating  to  a  flight  subphase  defined 
by  a  single  aircraft  state  even  though  the  task  occurs  in  other  sufcpha.ies  with 
diff  rent  aircraft  states. 

If  ar  evaluation  is  to  apply  to  a  flight  phase,  the  rating  will  tend  to 
be  weighted  by  the  more  adverse  subphase  characteristics .  On  the  other  hand . 
a  longer  task  representative  of  one  or  more  flight  phases  will  provide  a 


better  opportunity  for  the  pilot  to  assess  the  workload  involved ,  the 
adequacy  of  the  information  displayed,  and  the  effect  of  unanticipated 
environmental  disturbances . 

Short  term  maneuvers,  particularly  if  determined  to  be  the  critical  part 
of  the  task,  will  be  most  useful  when  the  effect  of  specific  stability  and 
control  parameters  are  assessed,  but  their  use  to  the  exclusion  of  a  long-term 
mission-oriented  task  can  reduce  both  confidence  xn  and  the  fidelity  of  the 
handling  qualities  evaluation.  The  inclusion  of  auxiliary  tasks  .in  a  a  evalua¬ 
tion  task  will  depend  upon  the  program  objective,  but  may  not  be  required  for 
handling  qualities  studies  in  which  the  control  task  is  of  primary  interest . 
Additional  workload  imposed  by  the  auxiliary  tasks  can  actually  interfere  with 
the  pilot's  evaluation  of  the  characteristics  of  greatest  interest  but ,  of 
coarse,  must  ultimately  be  taken  into  account . 

No  hard  and  fast,  rule  can  be  given  for  defining  the  evaluation  task 
because  it  is  obvious  that  the  program  objectives  will  play  a  significant 
part,  but  the  nearer  the  selected  task  represents  all  the  demands  of  the  real 
mission  during  the  flight  phase  or  subphase  buing  evaluated,  the  less  will  be 
the  extrapolation  required  of  the  pilot. 

For  pilot  ratings  that  apply  to  a  complete  fight,  flight  phase,  or 
subphase,  the  rating  must  consider  all  disturbances  specified  for  the  mission 
in  the  senre  of  being  required  operations.  Thus,  more  extrapolation  is 
expected  foi  these  ratings,  placing  greater  reliance  on  the  pilot's  judgement 
than  for  either  task  or  composite  ratings.  On  the  other  hand,  task  and  com¬ 
posite  ratings  must  be  reviewed  with  caution,  as  they  m«.v  not  include  the  most 
critical  conditions  and,  therefore,  could  be  misleading. 

The  selection  of  a  complete  fight,  flight  phase,  or  subphase  as  the  unit 
to  he  rated  will  therefore  depend  upon  the  program  objective,  the  character¬ 
istics  of  primary  interest,  and  the  degree  to  which  it  is  desired  to  draw  upon 
the  pilot's  training,  experience,  and  knowledge  to  assess  and  extrapolate 
beyond  tk  i  ^ecific  task  provided.  Several  aspects  of  task  selection  involved 
in  actual  practice  are  discussed  further  in  the  following  paragraphs. 

Task  's .  flight  phase  or  composite  ratings  To  provide  a  rating  based  on 
the  simulated  task  is,  of  course,  the  most  direct  approach  and  reduces  the 
uncertainties  of  as^’ng  the  pilot  for  extrapolation.  It  is  weak,  hovever,  in 
that  the  task  selected  may  not  represent  critical  flight  conditions,  thereby 
allowing  a  nonconservative  leniency  in  the  interpretation  of  the  pilot  rating. 
Obviously,  the  pilot  wil,  not  be  as  influenced  by  mission  objectives  if  he  is 
told  to  confine  his  rating  to  the  evaluation  task,  although  it  may  be  diffi¬ 
cult  foi  him  to  exclude  considerations  that  he  knows  he  would  consider  under 
actual  operating  conditions .  If,  however,  al 1  the  separate  tasks  or  subphases 
needed  to  define  a  flight  phase  can  be  provided,  separate  ratings  may  be 
obtained  for  each  task  or  subphase,  and  c  composite  rating  then  given  for  the 
multiple  tasks  or  lbphases  required  io  represent  a  flight  plane.  Providing 
all  the  separate  tasks  required  to  define  a  flight  phase  howex er,  is  seldom 
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possible  even  by  simulation.  Nor  is  it  always  possible  in  actual  flight, 
although  its  importance  may  dictate  flying  to  remote  areas  to  find 
environmental  conditions  for  the  actual  tests. 

If  it  is  then  not  possible  to  provide  all  the  separate  tasks  required  to 
define  a  given  flight  phase,  it  may  be  desirable  to  base  a  flight  phase  rating 
largely  on  the  pilot's  ability  to  draw  on  his  knowledge,  experience,  and  use 
of  pi  lot -induced  disturbances  or  self-induced  tasks.  Such  a  procedure 
utilizes  the  maximum  capability  of  the  pilot  but  also  introduces  extrapolation, 
with  the  inherent  risk  of  his  occasionally  overlooking  a  potential  problem  or 
of  being  unduly  critical  of  certain  characteristics.  The  extent  to  which  the 
pilot  is  expected  to  compensate  for  limitations  in  the  simulated  situation 
must  be  clearly  understood  at  the  start  of  a  program. 

If  the  time  factor  and  probability  of  occurrence  are  the  same  for  each 
task  the  composite  rating  will  likely  reflect  the  most  critical  task..  If  the 
probability  of  occurrence  is  low  for  one  of  the  tasks  (as  might  be  attribut¬ 
able  to  severe  turbulf nee) ,  it  is  still  possible  for  the  probability  of 
encounter  to  influence  a  composite  rating  in  an  undefined  manner  unless  such 
a  task  is  excluded  from  the  composite  rating.  The  experimenter  must  decide 
whether  composite  ratings,  which  include  "probability  of  failure"  or  probabil¬ 
ity  of  encountering  "unusual"  environmental  conditions,  are  to  be  assessed  by 
the  pilot  or  someone  else.  There  must  be  no  doubt  as  to  whose  responsibility 
this  is,  however. 

Transient  disturbances.-  When  a  steady-state  failure  is  designated,  the 
pilot  is  exempt  from  considering  the  probability  of  a  failure.  However,  some¬ 
one  must  consider  the  probability  of  encountering  transient  disturbances  from 
engine  or  other  failures,  wind  shear  or  other  short-term  environmental  condi¬ 
tion-'-  In  these  cases,  the  pilot  might  have  to  consider  the  duration  of  the 
disturbance  as  well  as  its  level.  If  specified  in  the  mission  definition,  the 
pilot  must  consider  disturbances  whether  they  are  provided  in  the  task  or  not. 

Admittedly,  a  rating  that  reflects  extrapolation  for  the  effect  of  an 
undefined  level  and  extent  of  a  disturbance  represents  an  approximation,  but 
this  may  often  be  re-.erable  to  ignoring  the  disturbance. 

T'  ?  effect  of  certain  disturbances,  such  as  those  induced  by  the 
environment  and  ttie  dynamics  of  failure,  on  pilot  rating  will  depend  heavily 
up<  the  aircraft  state.  As  a  rule,  transient  disturbances  from  whatever 
source  will  affect  the  pilot  rating  less  if  the  handling  qualities  are  good 
rather  than  poor.  For  this  reason,  it  becomes  increasingly  important  to  pro 
vide  more  accurate  dynamic  representation  of  transient  disturbances  as 
handling  qualities  deteriorate. 

Sin£_le  axis  or  single  parameter  evaluations .  -  Confusion  can  occur  when 
ratings  are  requested  for  specific  parameters  or  for  a  single  axis  rather  than 
for  the  complete  pilot  vehicle  combination  in  performance  of  its  mission.  It 
should  be  handling  qualities  that  are  rated,  with  the  influence  of  specific 
parameters  identified  through  changes,  if  any ,  in  the  pilot  ratings  and  from 
t h e  P i  1  o t  c o m me nt  dat  a 
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Single-axis  simulations  can  be  of  considerable  value  in  quick  look 
evaluations  but  should  be  clearly  identified,  so  that  the  limitations  of  the 
data  can  be  recognized.  The  task  that,  for  example,  includes  freedom  for 
maneuvering  in  pitch  only  cannot  provide  information  on  either  total  pi  lot 
workload  or  control  coupling.  When  a  single  axis  or  single  parameter  is  of 
primary  interest,  it  is  preferable  to  incorporate  the  single  axis  characteris¬ 
tics  into  a  full  multiple-axis  (six  degrees  of  freedom)  task  so  that  the 
evaluation  can  be  made  in  the  full  context  of  handling  qualities. 

Deficiencies  evaluated  in  the  longitudinal  flight  mode  cannot  be  simply 
added  to  deficiencies  evaluated  separately  in  the  lateral -directional  mode. 

As  a  consequence,  the  most  critical  single  axis  task  cannot  necessarily  be 
assumed  to  be  the  most  critical  overall  pilot-vehicle  task. 

A  parallel  situation  arises  in  the  general  application  of  quantitative 
criteria  for  handling  qualities.  As  a  rule,  such  criteria  are  developed  for 
specific  parameters  with  all  other  parameters  at  a  satisfactory  level.  Such 
is  not  necessarily  the  case  in  real  life,  so  that  one  cannot  expect  to  find 
single  parameter  criteria  that  apply  accurately  to  all  aircraft  with  their 
various  roles,  configurations,  and  operating  environments. 

Preoccupation  with  variations  in  a  single  parameter  should  not  be  allowed 
to  distract  the  pilot  from  the  fact  that  his  ratings  apply  to  handling  quali¬ 
ties  as  a  whole,  for  the  task  (or  flight  phase)  in  context  of  the  aircraft 
role.  Assessment  of  single  parameters  or  single  axis  effects  can  be  very 
helpful  to  a  designer,  but  special  care  must  be  taken  in  the  development  and 
presentation  of  such  data.  Programs  of  such  limited  scope  are  likely  to 
introduce  considerable  pilot  extrapolation  with  a  resulting  low  level  of  con¬ 
fidence  so  that  in  some  cases  it  may  be  more  desirable  to  obtain  only  pilot 
comment  data  or  comparat ive  results  not  associated  with  a  handling  qualities 
rating. 

An  evaluation  may  often  involve  comparisons  or  relative  assessment  of 
specific  characteristics,  but  the  evaluation  pilot  should  not  become  so 
engrossed  in  comparisons  that  he  loses  sight  of  the  absolute  aspects  of  per¬ 
formance  and  workload  in  the  context  of  the  role  and  mission  requirements. 

The  use  of  an  entirely  separate  rating  scale,  specifically  for  relative  assess¬ 
ment  of  individual  stability  and  control  parameters,  has  been  considered  but 
abandoned  because  of  possible  confusion  introduced  by  a  second  scale. 


Pilot  Assessment  Considerations 

Pi  lot  rat ing . -  As  a  shorthand  representation  of  the  handling  qualities  of 
an  aircraft  in  tne  performance  of  a  defined  mission  and  task,  a  pilot  iiting 
will  be  meaningful  only  in  proportion  to  the  care  taken  in  developing  ,e  pro 
gram  (defining  objectives,  the  role  and  mission,  the  evaluation  task,  > hat  the 
rating  applies  to,  the  simulation  situation  and  extent  of  pilot  extrapolation 


23 


involved).  Unless  a  common  basis  is  established  and  any  criteria  used  are 
clearly  indicated,  one  cannot  expect  to  achieve  reliable  data  and  comparable 
ratings  among  pilots.  Large  disagreement  between  pilot  ratings  is  usually 
traced  to  incomplete  program  development. 

There  tends  to  be  some  disagreement  among  pilots  as  to  how  they  actually 
arrive  at  a  specific  numerical  rating.  Some  pilots  lean  heavily  on  the  speci¬ 
fic  rating  description  and  look  for  the  description  that  best  fits  their  over¬ 
all  assessment.  Other  pilots  prefer  to  make  the  dichotomous  decisions 
sequentially,  thereby  arriving  at  a  choice  between  two  or  three  ratings.  The 
decision  among  the  two  or  three  ratings  is  then  based  upon  the  adjective 
description.  In  concept,  the  latter  technique  is  preferable  since  it  empha¬ 
sizes  the  relationship  of  all  quality  decisions  to  the  aircraft  role  and 
mission  requirements.  With  the  final  version  of  the  revised  scale,  the  pilot 
decisions  shown  must  be  considered  in  order  to  define  the  category  boundaries. 

The  actual  technique  used  is  probably  somewhere  between  the  two  tech¬ 
niques  discussed  above  and  is  not  so  different  among  pilots.  In  the  past,  the 
pilot's  choice  has  probably  been  strongly  influenced  by  the  relative  useful¬ 
ness  of  the  descriptions  provided  for  the  categories  on  one  hand,  and  the 
numerical  ratings  on  the  other.  The  evaluation  pilot  is  more  or  less  continu¬ 
ously  considering  the  rating  decision  process  during  his  evaluation.  He  pro¬ 
ceeds  through  the  dichotomous  decisions  to  the  adjective  descriptors  enough 
times  that  his  final  decision  is  a  blend  of  both  techniques. 

Half  ratings  (e.g.,  rating  4-1/2)  generally  indicate  reluctance  of  the 
evaluation  pilot  to  assign  either  of  the  adjacent  ratings  to  describe  the  con¬ 
figuration.  Any  finer  breakdown  than  half  ratings  is  hardly  ever  justified 
since  any  number  greater  than  or  less  than  the  half  rating  implies  that  it 
belongs  in  the  adjacent  group.  Any  distinction  between  configurations 
assigned  the  same  rating  must  be  made  in  the  pilot  comments.  Use  of  the 
3-1/2,  O’- 1  / 2 ,  and  9-1/2  ratings  is  discouraged  because  they  represent  impor¬ 
tant  "boundary"  conditions  and  the  decision  as  to  which  category  is  selected 
may  be  significant  even  though  hard  to  make. 

As  a  general  rule,  pilot  ratings  and  comments  are  preferably  given  on  the 
spot  when  the  characteristics,  performance,  and  workload  are  fresh  in  mind. 

If  the  pilot  should  later  want  to  change  his  rating,  the  reasons  for  the 
change  may  be  of  interest.  In  some  cases,  an  attempt  should  be  made  to  repeat 
the  configuration  later  in  the  evaluation  program. 

l'i lots’  comments.-  The  use  of  a  rating  scale  considered  for  universal 
hand i  ng  qualities  application  leads  to  the  assumption  that  the  numerical 
pilot  ■ at i ng  can  represent  the  entire  qualitative  assessment.  Lxtreme  care 
must  be  taken  against  this  oversimplification  because  the  numerical  ratings  do 
not  constitute  the  complete  results  of  the  data  gathering  process. 

As  one  might  expect ,  the  evaluation  data  most  often  neglected  are  the 
pilots'  comments ,  either  because  they  are  not  recorded  or  because  they  are 
often  difficult  to  deal  with  because  of  their  qualitative  form,  and  perhaps 
their  bulk.  Often  the  nature  of  the  experiment  allows  the  pilot  neither  the 
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time  nor  the  opportunity  to  identify  the  cause  of  his  objections.  Ratings, 
without  the  pilot's  objections,  are  only  part  of  the  story.  Only  if  the  defi¬ 
cient  areas  can  be  identified,  cart  one  expect  to  devise  improvements  to  elimi¬ 
nate  or  attenuate  the  shortcomings.  The  pilot  comments  are  the  means  by  which 
the  identification  can  be  made. 

The  pilots'  objections  to  the  handling  qualities  arc  important  therefore 
to  the  airplane  designer  who  is  responsible  for  improving  the  handling  qual¬ 
ities  and  to  the  engineer  who  is  attempting  to  understand  and  use  the  pilot 
rating  data.  If  ratings  are  the  only  output,  the  engineer  has  no  real  way  of 
assessing  whether  the  objectives  of  the  experiment  were  actually  realized.  A 
pilot's  comments  supply  a  means  of  assessing  whether  his  objections  (which 
lead  to  his  summary  iting)  were  related  to  the  mission,  to  some  extraneous 
factor  in  the  execution  of  the  experiment,  or  to  his  inaccurate  interpretat ion 
of  various  aspects  of  the  mission.  For  pilots'  comments  to  be  most  useful, 
several  details  must  be  kept  in  mind. 

Generally,  pilots  should  comment  in  the  simplest  possible  language. 
Attempts  to  translate  the  observed  characteristics  or  responses  into  engineer¬ 
ing  terms  should  be  avoided,  unless  such  terms  accurately  supplement  the 
pilot's  observations  given  in  descriptive  terms.  The  pilot  should  report  what 
he  sees  and  feels,  and  describe  his  difficulties  in  carrying  out  whatever  he 
is  attempting.  It  is  then  important  for  the  pilot  to  relate  his  difficulties 
in  executing  specific  tasks  to  their  effect  on  the  accomplishment  of  the 
required  operations. 

The  pilot  should  be  encouraged  to  make  specific  comments  when  evaluating 
each  configuration.  These  comments  generally  are  in  response  to  questions 
developed  during  discussions  of  the  mission  and  simulation  situation.  The 
pilot  must  also  be  encouraged  to  comment  regarding  his  difficulties  over  and 
above  the  answers  to  the  specific  questions  asked  of  him.  In  this  regard,  the 
test  pilot,  should  strive  for  a  balance  between  a  continuous  running  commentary 
and  only  occasional  comment  in  the  form  of  an  explicit  adjective.  The  former 
often  requires  so  much  editing  to  find  the  substance  that  it  is  often  ignored, 
while  the  latter  may  add  nothing  to  the  numerical  rating  itself. 

The  pilot's  comments  must  be  collected  during  or  immediately  after  each 
evaluation.  If  the  comments  are  left  until  the  conclusion  of  the  evaluation, 
they  are  often  forgotten.  For  both  in-fl;ght  and  ground  simulator  evaluations, 
this  means  that  provision  should  be  made  for  wire  or  tape  recording.  The 
immediate  recording  of  key  words  or  phrases  on  a  kneepad  can  often  be  easily 
expanded  to  the  full  content  after  the  task  or  evaluation  is  completed.  It 
has  been  the  experience  of  the  authors  that  the  best  voluntary  comments  are 
often  given  during  the  evaluation.  A  useful  procedure  is  to  encourage  volun¬ 
tary  comment  and  note -taking  during  the  evaluation  and  to  require  answers  to 
specific  questions  in  the  summary  comments  at  the  end  of  the  evaluation. 

Questionnaires  or  explicit  check  lists  ensure  that:  (a)  all  important  or 
suspected  aspects  are  cor.-  idered  and  not  overlooked,  (b)  the  reason  for  a 
given  rating  is  specified,  ( c )  an  understanding  is  provided  of  the  tradeoffs 
with  which  pilots  must  continually  contend,  and  (d)  supplementary  comment  that 
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Figure  8.-  Classification  of  PIO  tendency. 


might  not  be  offered  otherwise  is 
stimulated.  It  is  recommended  that 
the  pilots  participate  in  the  prepar¬ 
ation  of  the  questionnaires.  The 
questionnaires  should  be  modified  if 
necessary  as  a  result  of  the  pilots' 
initial  evaluations.  On  occasion,  it 
may  be  desirable  to  classify  pilots' 
comments  by  having  the  pilots  select 
one  of  several  ranked  comments  about 
a  specific  characteristic.  An  exam¬ 
ple  of  such  a  classification  of  spe¬ 
cific  pilots'  comments  is  shown  in 
figure  8  for  PIO  tendency.  Classifi¬ 
cation  as  shown  in  this  example  is 
for  easy  identification  only,  and  is 
net  to  be  confused  with  the  designa¬ 
tion  for  pilot  rating. 


Simulation  Situation 

A  pilot  evaluation  is  seldom  conducted  under  the  full  circumstances  of  an 
actual  flight.  It  almost  always  involves  simulation  to  some  degree  because  of 
the  absence  of  the  real  situation.  As  an  example,  the  evaluation  of  a  day 
fighter  is  seldom  carried  out  under  the  circumstances  of  a  combat  mission 
where  the  pilot  is  not  only  shooting  at  real  targets  but  is  being  shot  at  with 
real  bullets.  Therefore,  after  the  program  has  been  defined  and  the  decision 
made  as  to  what  is  being  rated,  the  relationship  of  the  simulation  to  the  real 
situation  must  be  explicitly  stated  for  both  the  engineer  and  the  evaluation 
pilot  so  that  each  may  clearly  understand  the  limitations  with  respect  to  what 
is  provided  in  the  task,  the  environmental  disturbances,  the  cockpit  interface, 
and  the  completeness  of  the  simulation  of  the  stability  and  control 
characteristics . 

The  pilot  and  engineer  must  know  not  only  wnat  is  left  out  of  an 
evaluation  program,  but  also  what  As  in  that  should  not  be  in.  The  fact  that 
the  anxiety  and  tenseness  of  the  real  situation  are  missing,  and  that  the  air¬ 
plane  is  flying  in  the  clear  blue  of  calm  daylight  air  instead  of  in  the  icy, 
cloudy,  turbulent,  darkness  of  the  real  mission  could  affect  the  results  and 
should  be  considered  during  the  program  development  and  the  interpretation  of 
the  results  . 

Cockpit,  interface.-  The  cockpit  interface  must  be  specified,  at  least  to 
the  extent  necessary  to  identify  any  factors  that  influence  handling  quali¬ 
ties.  As  a  rule,  once  this  is  done  for  a  given  program  it  remains  fixed  and 
is  merelv  reference  information.  When  stability  and  control  characteristics , 
for  example,  are  the  primary  interest,  features  pertaining  to  the  flow  of 
information  to  and  from  the  cockpit  probabl)  would  not  be  cons i de reo  as  vari¬ 
ables  in  the  study,  but  they  must  be  rioted  with  sufficient  detail  to  establish 
at  least  any  possible  qualitative  influence  on  the  results. 
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The  individual  cockpit  features,  such  as  control  system  characteristics 
and  instrument  display,  may,  however,  often  include  the  specific  variables  to 
be  studied.  In  some  cases,  these  variables  will  be  made  known  to  the  evalua¬ 
tion  pilot,  while  in  others,  it  will  be  more  important  that  he  assess  only  the 
overall  handling  qualities  with  respect  to  task  accomplishment. 

Unless  control  system  characteristics  are  a  variable  to  be  studied,  it  is 
desirable  to  provide  an  accurate  representation  of  the  anticipated  vehicle 
control  system.  Such  factors  as  control  system  break-out  force,  friction, 
inertial  characteristics,  or  system  gearing  and  sensitivity  may  significantly 
affect  pilot  rating.  If  it  is  not  practical  to  represent  the  vehicle  control 
system  characteristics  accurately  then  system  characteristics  should  be  pro¬ 
vided  that  are  at  least  satisfactory  to  the  evaluation  pilots  and  do  not 
introduce  unwanted  problems.  It  is  often  desirable  to  allow  evaluation  pilots 
to  establish  comfortable  levels  of  control  sensitivity,  for  example,  before 
beginning  an  evaluation  program.  This  procedure  has  distinct  advantages  in 
reducing  the  number  of  uncontrolled  variables  that  may  enter  into  an  evalua¬ 
tion  program.  Care  must  be  taken,  however,  that  this  procedure  is  not  used 
when  the  control  sensitivity  is  a  variable. 

Repositioning  any  of  the  selectors  available  to  the  flight  crew  to  change 
aircraft  configuration  will  depend  on  the  flight  phase  or  task  being  evaluated. 

Visual  .information  available  to  the  pilot  is  defined  by  such  features  as  the 
cockpit  cut-off  angle,  the  simulator  system  for  providing  external  information, 
and  the  types  and  locations  of  the  cockpit  instruments. 

Aircraft  environment.-  The  extent  to  which  aircraft  environmental  factors 
are  included  in  the  simulation  must,  be  described.  It  is  important  that  the 
level  of  such  disturbances  be  related  to  the  length  of  time  they  are  to  be 
coped  with  This  is  usually  for  the  duration  of  a  flight  phase,  or  specified 
task,  but  if  it  is  to  be  considered  as  a  transient  condition,  this  should  be 
noted . 

Should  the  evaluation  be  conducted  under  actual  flight  conditions,  it  may 
be  desirable  to  secure  a  turbulence  classification  based  on  the  standardized 
scale  from  the  FAA  Airman's  Information  Manual  (fig.  9),  or  to  relate  to  other 
documentation  as  may  be  available,  < 

The  probability  of  occurrence  of  various  levels  of  turbulence  is  an 
important  consideration,  but  is  much  too  complex  fcr  the  pilot  to  consider 
during  an  evaluation. 

The  influence  of  e  vironmental  disturbances  varies  inversely  with  the 
level  of  nandling  qualities;  that  is,  in  general,  the  poorer  the  handling 
qualities,  the  more  they  are  influenced  by  adverse  disturbances.  There  is  no 
simple  rule,  however,  for  translating  ratings  obtained  in  smooth  air  to 
ratings  applicable  to  rough  air. 

Hnvironmental  disturbances  with  some  undefined  probabilitv  of  occurrence 
can  strongly  influence  the  aircraft  design  and  its  handling  qualities.  lor 
this  reason,  their  inclusion  can  be  essential  to  the  conduct  of  handling  qual 
tries  experiments.  When  no  provision  is  made  for  actual  or  simulated 
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^Intensity 

Aircraft  reaction 

Reaction  inside  aircraft 

Reporting  term 

Definition  ^ 

Turbulence  thof  momentarily  causes  shght  changes  m  aircraft 
attitude,  a'tifude,  or  heoding  Repot  as  Light  Turbulence 

Occupants  may  feei  a  slight  sfrom  ogornsf 
seat  belts  or  shoulder  straps. 

Occasional 

Less  than  I/3 
of  the  time 

OR 

Unsecured  objects  remain  at  rest,  food 

- . . - 

LIGHT 

service  moy  oe  conducted,  and  bttie  or 

Inte.  rniMent 

'/}  ,r>  2/3 

Turbulence  (light  bumpiness)  that  causes  snght  aircraft 

no  difficulty  is  encountered  m  walking 

.  J 

fluctuations  of  rapid  intervals  without  appreciable  change  in 
altitude,  roll,  or  yaw.  Report  as  Light  Chop 

Continuous 

More  ’hen  2/j 

Turbulence  that  is  similar  to  Light  Turbulence  but  of  greater 

Occupants  feel  definite  strains  against 

NOTE  j 

intensity  Changes  »n  aircraft  attitude,  altitude,  or  heading 

seat  belts  or  shoulder  streps 

i .  Pilots  should  repo 

r  t  lot  af'Onis),  j 

nccur,  but  the  aircraft  remains  m  positive  control  at  an  times 

Unsecured  objects  ore  d.siodged 

time,  intensity,  altitude,  type  of 

Report  as  Moderate  Turbulence 

rood  service  and  waik-ny  are  difficu't 

ancroft  and,  when  applicable, 

MOOf  RATE 

OR 

duration  of  turbulence. 

Turbulence  that  'S  vm.ior  to  i -gnt  Chop  but  of  greater 

2.  Duration  may  be  based  on  t.me 
between  two  locations  o r  over  a 

^tensity  If  ccuses  rapid  bumps  or  jolts  without  appreciable 
Change  in  aircraft  altitude,  roll,  Of  yaw 

Report  as  Moderate  Chop 

smgie  location.  All  locations  should 
be  'eod'iy  identifiable 

EXAMPLES 

Turbulence  that  causes  large  changes  m  aircraft  altitude, 

Occupants  ore  forced  violently  and  ! 

a  Over  Omaha,  l<?3 

?Z.  moderate 

rvtiTude,  or  heading  It  may  cause  large  variations  m  -ndco’ea' 

repeatedly  ogams t  seat  belts  or  shoulder  J 

turbulence,  Plight 

Level  300, 

SEVERE 

airspeed  Aircraft  may  be  momentarily  out  of  control 

strops. 

8/07 

Repor»  as  Severe  Turbulence 

Unsecured  ot>i*cts  ore  tossed  about  1 

D.  From  SO  mtles  South  Qt  I 

Food  service  and  walking  are  impossible  j 

Albuquerque  to  30 

mtles  north  of  j 

EXTREME 

V _  J 

Turdjtence  which  tne  aircraft  is  violently  tossed  about  and 
is  practical  ly  mposs'ble  to  control  It  moy  cause  structural 
damage  Report  as  Enfreme  Turbulence 

i 

! 

Phoer.1*.  i£Xi£  to  li'bOrC. 

Kcosionoi  Moderate  Chop,  v  ngnt 
l  evel  3?0.  DC -6 

J 

Figure  9.-  Turbulence  criteria. 

environmental  disturbances,  it  may  be  necessary  for  the  pilot  to  learn  their 
effect  on  handling  qualities  by  employing  self-imposed  disturbances.  Such 
procedures  have  limited  application  in  a  sophisticated  simulation  program,  but 
do  have  considerable  value  in  "short  look"  programs. 

Pi  lot  stress . -  Surprise  and  stress  can  interfere  with  the  pilot's 
performance  by  distracting  him  from  the  primary  task.  Considering  these  fac¬ 
tors  in  depth  is  usually  left  to  the  more  sophisticated  programs,  but  those 
elements  that  contribute  to  greater  fidelity  and  realism  in  a  simulation  pro¬ 
gram  will  be  found  desirable.  The  sharp  jolt  of  the  cockpit  when  a  bad  land¬ 
ing  occurs  contrasts  vividly  with  the  smooth  conditions  of  a  good  landing. 

An  element  of  surprise  and  stress  should  be  introduced  when  the  dynamic 
effects  of  control  system  or  SAS  failures  are  being  evaluated.  To  be  truly 
objective  would  require  data  from  many  occurrences  introduced  in  a  random  man¬ 
ner  during  the  time  the  pilot  is  engrossed  in  the  performance  of  a  task. 

If  surprise  and  stress  are  not  included  in  the  task,  the  evaluation  pilot 
must  simulate  surprise,  distraction,  or  unattended  operation  by  using  a 
delayed  reaction  tune.  The  stress  or  rapid  increase  in  pilot  workload  when 
two  or  more  adverse  conditions  or  situations  occur  together  can  be  studied  in 
more  sophisticated  simulation  programs.  These  potential  "operating  problems" 
consisting  of  combinations  of  failures  or  weather  conditions  are  apt  to  raise 
the  question  ol  probability  of  occurrence.  Again,  this  question  should  be 
separated  from  the  pilot  assessment  whenever  possible.  The  pilot  rating 
should  reflect  the  difficulty  and  effect  on  performance  of  the  occurrence 
relative  to  successful  completion  of  the  task  or  flight  phase  under  evaluation. 


28 


Therefore,  such  occurrences  must  originate  with  the  program  objective  and  be 
documented  in  the  test  plan  as  to  how  they  will  be  executed;  that  is,  whether 
they  will  be  included  in  a  task  or  left  to  the  pilot's  individual  assessment. 

Pilot  extrapolation  and  confidence.-  The  question  of  simulation  enters 
into  nearly  every  evaluation  program  to  some  degree.  Previous  studies 
(ref.  9)  have  shown  that  sophistication  is  not  necessarily  the  key  to  simula¬ 
tor  usefulness  although  it  can  extend  the  range  of  application.  Deciding 
"what  a  pilot  rating  applies  to"  (specific  task  or  flight  phase),  and  the 
completeness  of  the  simulation  will  determine  the  degree  to  which  pilot 
extrapolation  is  to  be  relied  on.  Neither  the  pilot  nor  the  engineer  retains 
confidence  in  the  results  if  the  need  for  extrapolation  of  observed  results 
becomes  too  great.  Guidelines  have  been  drawn  in  this  report  to  guard  against 
this  problem  while  at  the  same  time  encouraging  full  utilization  of  the 
trained  and  experienced  test  pilot. 

It  is  felt  that  careful  planning  and  agreement  on  program  objectives, 
mission  definition,  what  is  being  rated,  and  the  execution  of  the  experiment 
can  limit  the  uncertainties  of  extrapolation. 


In  order  to  provide  a  means  for  expression,  on  the  part  of  pilots  in 
particular,  a  confidence  factor  could  be  introduced  into  programs  in  which 
simulation  is  involved.  A  definition  might  be  "the  ratio  of  the  information 
available  (to  the  pilot)  in  the  simulation  situation  to  the  information 
required  to  derive  a  realistic  pilot  rating."  While  provision  is  made  for  the 
use  of  such  a  confidence  factor,  no  recommendation  as  to  its  use  is  given  in 
this  report.  If  used,  care  must  be  taken  to  insure  that  the  confidence  factor 
does  not  inhibit  or  otherwise  confuse  the  actual  pilot  ratings  assigned.  As 
reference  information,  a  confidence  factor  could  help  a  pilot  focus  his  com¬ 


ments  on  the  adequacy  of  the  simulation, 
same  purpose. 

•  CLASS  A 

A  D' tot  may  usually  iss'Qr'  0  rating  w  »h  a  relatively  h.gh  degree 
of  co^f. Pence,  although  he  may  have  mild  reservations  because  of 
incomplete  or  inadequate  simukJt'on  of  motion  Cues,  disturbances, visual 
mforrnafon  ,  or  Other  foctors  o f  fee  1  mg  pilot  wOrkioOd 

Supplementary  tasws,  f  needed,  con  be  aceq>joreiy  provided  by 
the  pn'o* 

•  ‘.lA$S  b 

A  fiiot  con  ass' jr  a  rating  with  only  a  moderate  ieve<  of 
confidence  bee  ajse  u  H'erjjinhef.  introduced  by  a  'or  *  of  representative 
environmental  ci star  bonnes  is  we'  as  incomplete  inadequate  simu|otio.‘’ 
uf  motion  vies.  I  s'..' bon(  es.  v  VjCJI  informat  on.  or  Ofher  fac»v  rs 
a  *  tec  f  mg  >  *;•'  v  id 

Suppte -mentor  y  (**  dev»ed,  but  o-e  ■  O’  Ov\MOb'e 

•  i  a  '  ■>  s  c: 

A  pnof  .  an  assign  a  Mt-r-g  win  miy  inn  ■rm,- n-  c  ontioer' .  e  beiduse 
.  ■«,side*ab'p  poo*  e»'nipi.!,ii  n  >.  <r  p-  pd  due  *a  .v>  .-mpiete  tosh, 
rtve-ei'y  rpq,.  ’  '  t  .'\«Wohip  rf  :\m»  e  :  r-  serf  iiT«p.i>ed  tasks  and 
iTH)ne  .i  vi'i  in  assessment 

This  may  also  lie  aggravated  b>  -ncoret  =e»e  ■  >  »c ,  cm.ted 
Mfn.natic.rs  of  'iiolti if  i  lies ,  .fisru't'ijn*.  es .  vvjip  ’■  u  nijton  .  >r 


but  questionnaires  could  serve  the 

For  those  who  favor  using  a 
confidence  factor,  a  simple  classi¬ 
fication  is  proposed  in  figure  10, 
as  A,  B,  or  C.  Studies  involving 
"quick  look"  or  rudimentary  simula¬ 
tion  may  often  call  for  pilot  rat¬ 
ings  in  which  the  pilot  is  confident 
in  only  a  relative  rather  than  an 
absolute  sense.  For  these,  a  "C" 
classification  might  be  appropriate. 
An  attempt  should  be  made  to  place 
all  other  evaluations  in  Class  A, 
thereby  allowing  Class  B  to  be  used 
only  when  a  pile  has  di  i'f  i  cul  ty 
deciding  between  A  and  C. 


# 


f  1  ^  Li  ?  «.*  10 


1  as  S  l  f  1  i  .if  Lull  r!  pilot  .  ciltulcncc  i.ii  till 


The  Briefing  Guide 


In  order  to  summarize  the  content  of  this  report  in  a  form  which  may  be 
applied  directly  to  the  execution  of  handling  qualities  experiments,  a 
"Briefing  Guide"  has  been  assembled  in  appendix  B.  The  purpose  of  this  guide 
is  to  outline  a  format  to  insure  that  all  pertinent  documentation  is  covered 
for  each  evaluation  program.  Only  in  this  way  can  the  evaluation  pilots  be 
sure  that  everyone  is  talking  about  the  same  thing.  Subsequent  analysis  of 
the  data,  reporting,  and  ultimate  comparison  with  data  from  other  sources  is 
then  materially  aided. 

The  briefing  guide  enables  the  pilot  and  engineer  to  know  what  is  missing 
from  an  evaluation  program,  as  well  as  what  is  provided. 

Once  the  information  is  tabulated,  it  should  be  apparent  just  what  is 
being  evaluated  and  rated,  what  the  task  is,  the  conditions  under  which  the 
evaluation  is  performed,  and  the  applicability  of  the  data.  With  the  impor¬ 
tant  information  provided  in  a  Briefing  Guide,  it  should  be  possible  to 
evaluate  handling  qualities  mere  effectively  and  to  improve  communication 
not  only  between  pilots  and  engineers,  but  between  various  research  and 
development  groups  as  well. 
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COMPENSATION  - 


CONFIGURATION  - 


COCKPIT  INTERFACE  - 


CONTROLS  - 


Primary  - 


Secondary  - 


Selectors  - 


APPENDIX  A 

DEFINITIONS 


The  measure  of  additional  pilot  effort  and  attention 
required  to  maintain  a  given  level  of  performance  in  the 
face  of  deficient  vehicle  characteristics. 

The  aircraft  geometry  as  established  by  the  actual 
position  of  movable  portions  and  surfaces  controllable 
by  the  selectors,  and  the  state  of  operability  of 
on-board  systems . 

Examples:  Auto- thrott le  on,  flaps  at  15°,  wing  sweep 
at  45°,  etc. 

The  means  provided  for  the  flow  of  information  to  and 
from  the  pilot.  These  include  the  display  of  informa¬ 
tion  available  to  the  pilot  as  well  as  the  type  and 
characteristics  of  the  cockpit  controls. 

Examples:  Description  of  cockpit  instruments  and  layout 
Description  of  control  system  (i.e.,  stick, 
wheel,  force  -  deflection  gradients,  sensi¬ 
tivities,  primary,  secondary  controls,  and 
selectors  provided). 

A  distinction  is  made  between  the  types  of  controls  in 
the  cockpit  according  to  their  function.  Principal  con¬ 
trols  are  the  primary  and  secondary  controls. 

Those  controls  used  by  a  pilot  to  continuously  modify 
the  movement  of  the  aircraft. 

Examples:  Pitch,  roll,  yaw  controls,  tnrottle,  DEC. 

Those  controls  used  by  a  pilot  to  make  discrete  changes 
in  the  movement  or  balance  of  the  aircraft,  thereby  modi¬ 
fying  the  need  for  actuation  of  the  primary  controls. 

Examples:  Pitch,  roll,  and  yaw  trimmers,  aerodynamic 

braking  devices. 

Those  cockpit  controls  available  to  the  crew  for 
changing  aircraft  configuration. 

Examples:  Flaps,  slats,  wing  sweep,  BLC. 
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FAILURE  STATE  - 


FLIGHT  or  SORTIE  - 


FLIGHT  PHASE  - 


FLIGHT  SUB PHASE  - 


A  steady-state  failure  characterized  bv  the  various 
failed  systems  that  aifect.  the  handling  qualities.  The 
dynamic  effect  of  a  fai1  re  is  c,.,ilcd  a  change  of  state 
and  should  be  noted  separately. 

Examples:  Any  failure  resulting  in  loss  of  selected 

function.  ne  failure,  augmentation  sys¬ 

tem,  failure  in  stability,  autothrottle,  pri¬ 
mary  flight  control  system  (power  boost, 
electric  stick,  servo  control  •‘Tel,  etc.)  or 
secondary  flight  control  system  (trim,  aero¬ 
dynamic  brake,  etc.). 


A  complete  sequence  of  flight  phases  of  an  airciaft 
within  one  of  its  roles.  Full  or  complete  mission. 


Example:  The  composite  of  takeoff,  climb,  cruise,  com¬ 

bat  (or  other  special  phase!,  descent, 
approach,  landing. 


A  designated  portion  or  segment  of  a  complete  flight.  A 
mission  phase.  A  flight  phase  may  be  represented  by  one 
or  more  separate  tasks. 


Examples:  (a)  1  onunor  phases  -  takeoff,  climb,  cruise, 

descent,  approach,  and  landing. 

(b)  Speci  .  phases  required  by  role  - 
formation,  refueling,  air-to-air  or  ail  to 
ground  combat,  weapon  delivery,  emergency 
conditions  (i.e.,  2-  or  3 -engine  operation, 
emergency  descent,  etc.),  VTOL  transition, 
VTOL  hover,  STOL  takeoff,  and  STCL  approach. 

That  part  of  a  flight  phase  having  a  single  objective, 
and  a  single  configuration  or  change  in  a  configuration. 

Examples:  Air-to-air  tracking,  terminal  area  holding, 

glide  slope  capture,  localizer  capture,  I LS 
tracking,  wave-off. 


Those  qualities  or  characteristics  of  an 
govern  the  ease  and  precision  with  which 
to  perform  the  tasks  required  in  support 
role. 


aircraft  that 
a  pilot  is  able 
of  an  aircraft 


HANDLING  QUALITIES  - 


3° 

MANEUVER  - 

MISSION  - 

PERFORMANCE  - 

ROLE  - 

SPECIAL  CONDITIONS  - 

STATE  - 

TASK  - 

Control  - 


A  planned  and  regulated  Movement  of  an  aircraft  fox  the 
purpose  of  aiding  the  completion  of  a  given  control  task. 

Examples:  Bank,  turn,  dive,  pullup,  turn  reversal,  roll 

ltversal,  rolling  pullup,  steady  sideslip, 
return  from  sideslip,  control  steps  and 
pulses,  maintenance  of  a  steady  condi  t.:  on. 

The  ccmnosite  of  pilot-vehicle  functions  that  must  be 
performed  to  fulfill  operational  requirements.  May  be 
specified  for  a  role,  complete  flag  flight  phase,  or 
flight  subphase. 

The  precision  of  eortrd  with  respect  to  aircraft  move¬ 
ment  that  a  pilot  i:  able  to  achieve  in  performing  a 
task  (Pilot-vehicle  performance  is  a  measure  of 
handling  performance.  Pilot  performance  is  a  measure 
of  the  manner  or  efficiency  with  which  a  pilot  moves 
the  principal  controls  in  performing  a  task.) 

The  function  or  purpose  that  defines  the  primary  use  of 
an  aircraft 

The  special  circumstances  pertinent  to  the  evaluation 
(i.e,,  aircraft  environment  and  pilot  stress). 

Examples:  Sped'’  conditions  of  weather  and  environment, 

turbulence,  wind  shear,  ceiling,  visibility  - 
night,  etc.  Pilot  awareness,  surprise,  or 
distraction  with  respect  to  impending  failure 
or  disturbances. 

The  mass  distribution  and  failure  situation  that  deter¬ 
mine  completely  the  behavior  characteristics  of  the 
aircraft.  A  state  without  a  failure  is  a  normal  state. 

The  actual  work  assigned  a  pilot  to  be  performed  in 
completion  of  or  as  representative  of  a  designated 
flight  segment . 

That  part  of  a  task  which  requires  continuing  actuation 
cf  the  principal  controls  and  use  of  the  selectors  (see 
"CONTROLS")  as  required. 

Examples:  Movement  between  specified  points,  trading 

part  of  weapon  delivery,  I LS  or  "OR  tracking. 
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Auxiliary  - 


WORKLOAD  - 


Phy  si  cal 


Mental  - 


That  part  of  a  task  which  involves  the  pilot  in  actions 
other  than  direct  control  of  the  aircraft. 

Examples:  Navigation,  communication  monitoring,  and 
selection  of  systems. 

The  integrated  physical  and  mental  effort  required  to 
perform  a  specified  piloting  task. 

The  effort  expended  by  the  pilot  in  moving  or  imposing 
forces  on  the  controls  during  a  specified  piloting  task. 

Mental  workload  is  at  present  not  amenable  to  quantita¬ 
tive  analysis  by  other  than  pilot  evaluation,  or  indi¬ 
rect  methods  using  physical  workload  (inputl  and  the 
task  performance  measurements.  An  example  would  be  the 
improvement  associated  with  flight-director  type  dis¬ 
plays  which  reduce  the  mental  compensation  normally 
required  of  the  pilot. 
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APPENDIX  8 


BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 


PROGRAM  DEFINITION 


Objectives 

General : 

Research^]  Development . Q]  Acceptance  Q]  Other |  | 

Aircraft  role _ _ 

Flight  segment  of  interest _ 

Parameter  or  variable  of  primary  interest _ _ 

Mission  description: 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1,  What  is  pi lot-vehi cle  combination  required  to  accomplish? 


2.  What  are  the  conditions  under  which  these  required  operations  are 
to  be  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
interface) ? 
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Scope 

Short  look  (guide  to  further  tests) Q 
Long  look  (definitive  as  possible)  f~) 

Task  related  only  (minimize  pilot  extrapol ation) Q 

Mission  or  flight  segment  related  (pilot  extrapolation  encouraged) j  j 
Nature  of  task  or  tasks  provided: O 
Short  term  Q  Long  term  Q 

Critical  task.(s)  included? Q  Or  left  for  pilot  extrapolation?  |~1 
Identify: _ _ _ _ 


Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 


Use  of  pilot-initiated  or  self-imposed  tasks,  disturbances,  or  short¬ 
term  maneuvers. 


Provision  for  familiarization: 


Measured  performance: 

Is  performance  to  be  measured? 
Pi  lot- vehicle  (output ) |  | 

Pilot  workload  ( i  nput )  [~~1 
Other  measurements? Qj 


Additional  provisions: 


Are  additional  runs  allowed  for  evaluation? 


Explain: 


RATING  INFORMATION 


What  is  to  be  rated? 


[^Handling  qualities  |  |  Other 


I I  Flight  segment  (flight  phase  or  subphase) 


!  | Composite  of  tasks  (specify  tasks  included  in  composite  rating) 


I  I  Composite  of  subphases  (specify  subphases) _ 


What  is  provided? 


Test  vehicle:  (aircraft,  spacecraft,  simulator,  etc.) 


Aircraft  state: 


Normal  I  I  Emergency  I  I  Identify  failure: 


Conf igurat ion : 


Gross  weight: 


Mass  distribution: 


Changes  in  aircraft  state: 


Configuration  changes  (during  task  or  designated  phase) 
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Transient  failures  (unanticipated  failures  to  be  introduced  into  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 


Cockpit  interface: 

(Brief  description  of  important  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls: 
Primary  controls: 


Secondary  controls: 


Selectors : _ 

Cues  and  disturbances  provided: 


Mot i t  n . 


Vi sua 1 : 

I  ns t  ru  nent  s  :  Convent  1  ona  1  [23  Nave  1  23 

Fui 1  panel  23  Par:  panel  23 

(Brief  statement  of  instruments  or  characterist ics  which  mav 
af feet  r  be  eva 1  at i on ! 
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External  vision: 

External  visual  display  VFR  provided Q  not  provided  Q 

IFR  only  □ 

(Brief  description  of  information  provided) 


Task (s) : 

Control  task  description: 


Auxiliary  tasks?  Yes  Q]  No  j  | 

Ai rcraft  environment : 

Day  □  Night  □ 

Visibility  conditions: 

Turbulence  j  j 
Wind  shear  J 
Crosswind  I  I 


Other  weather  conditions: 
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PILOT  EVALUATION  DATA 

Form  of  results 

Pilot  rating  Qj  Comments  l]]  Questionnaire  Q  Oral  debriefing  |  | 
Other  Q _ _ _ _ 

Pilot  rating  scale: _ _ _ 

Principal  items  for  comment: _ _ _ _ 

Program  assessment: 

(Comments  and  recommendations  on  validity,  effectiveness,  etc. 

Confidence  factor  -  if  used) 


APPENDIX  B  -  Example  1 


BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 

PROGRAM  DEFINITION 

Objectives 

General : 

Research  □  Development  S3  Acceptance  I _ I  Other  □ 

Aircraft  role  Commercial  transport 

Flight  segment  of  interest  Landing  approach _ _ 

Parameter  or  variable  of  primary  interest  L  ,yigi  tudi nal  s  tab ility 
Mission  description: 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1.  What  is  pilot-vehicle  combination  required  to  accomplish? 

Conduct  VFR  and  IFR  approaches  and  landings ,  either  Dag  or  Might, 

with  an  accuracy  ±50  ft  from  desired  path  and  ±5  k  airspeed.  Ninety 

percent  consistency  is  allowable  if  pilot  can  execute  wave  off  safely 

or  make  an  acceptable  landing.  (5  ft/sec  at  touchdown ,  within  50  ft 

of  centerline,  with  side  velocity  less  them  5  ft /sec) 


2.  What  are  the  conditions  under  which  these  required  operations  are 
to  be  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
i nterface ) ? 

Both  normal  and  emergency  state  with  pitch  NAS  failed.  Approach 
configuration Light -Moderate  turbulence ,  Night  or  Day,  0-0.0  k 
crosswind,  10  k  per  100  ft  windshear.  Cockpit  interface 
items  must  not  be  considered  limiting  or  to  introduce  significantly 
object  i  enable  characteristics  to  the.  evaluation  pilots. 
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Scope 

Short:  look  (guide  to  further  tests)  |  ] 

Long  look  (definitive  as  possible)  |X' 

Task  related  only  (minimize  pilot  extrapolation)  Cj 

Mission  or  flight  segment  related  (pilot  extrapolation  encouraged) [X] 
Nature  of  task  or  tasks  provided: 

Short  term  Cj  Long  term  [Xj 

Critical  task(s)  included?  LJ  Or  left  for  pilot  extrapolation?  §C) 

I  dent i f y :  Consider  effect  of  higher  level  of  turbulence  on  instrument 
approach . _ _ 


Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 

high t  turbulence  and  wind  shear  provided. _ _ 

Use  of  pi  lot- ini tiated  or  self-imposed  tasks,  disturbances,  snort  - 
term  maneuvers. 

During  familiarization  runs  only. _ _ _ 


Provision  for  familiarization:  F'~>e  runs  at  beginning  of  eacn _ 

session. _ _ _ _ _ 

Measured  performance 

Is  performance  to  be  measured.'  Yes _ _ _ _ 

Pi  lot-vehi  cle  (output)  |3§  ILC  tracking  error  _ _ _ 

7  'on  trot  column  an’T  iTirottu?  aotivi  tu 
Pilot  workload  (input)  !59  ( reversals  <:n.<  vor'd 

Other  measurements ?  |X]  Altitude  and  cerspeed  error 


) 


Are  additional  runs  allowed  for  evaluation?  IVr 


lixp  lain: 


Pilot  may  make  nonperformance  evaluation  rum  an  time  allown  after 


comp  lotion  of  all  data  runs. 


RATING  INFORMATION 

What  is  to  be  rated? 

ES  Handling  qualities  CD  Other _ _ _ _ 

□  Task _ 

!X}  Flight  segment  (flight  phase  or  subphase)  Landing  approach 
CD  Composite  of  tasks  (specify  tasks  included  in  composite  rating) 

Composite  of  subphases  (specify  subphases) _ _ 

What  is  provided? 

Test  vehie'e:  (aircraft,  spacecraft,  simulator,  etc.)  S-16  Moving 
Base  Simulator 


Aircraft  state: 

Normal  CD  Emergency  [X|  Identify  failure:  Pitch  SAP  failed. 

Conr igurat  i  on  :  Approach  confi  gura  t  ion ,  gear  t  ,  fl  up  i  40" ,  Ira,. 
c'/.io  •  t'xt  ended,  i  a  r.w.vr  ,  etc. 

Ci  v  we  ight  :  Maximum  Priding  weigh  f 
Mass  distribution:  ■ at  aft.  !:mit 
Changes  in  a  i  malt  tate: 

t.'onf  i  gurat  i  on  changes  (during  task  or  designated  phase) 

::  efiee'i  ■>:  ;  fr  ••  ••  .’.o’er  i:  <  r,  ;  .<• 
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Transient  failures  (unanticipated  failures  to  he  introduced  into  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 


Pitch  SAS  failure  to  be  introduced  unexpectedly  on  two  5 AS  on  runs 


Cockpit  interface: 


(Brief  description  -•«'  important  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls: 


Primary  controls:  Conventional  wheel  and  yoke  with  707  feel 


characteristics ;  single -throttle  control  for  all  engines. 


Secondary  controls:  Electric  trim  switch  on  wheel  for  pitch  and 


roll,  yaw  trim  -  not  provided;  spoiler  actuation  not  provided  in 


cockpit,  but  available  on  voice  command. 


Selectors : 


Cues  and  disturbances  provided:  Pitch  an!  roll  cues  with  turbulence 


in  vertical  motion. 


Mot i on : 


V i sum  1 


Instruments:  Conventional  Xl  Novel 


lu i 1  pane  1 


Part  panel  X 


(Brief  statement  of  instruments  or  eh  a rao t e r i s t  i es  which  mav 


affect  t  he  eva luat i on ) 
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External  vision: 

External  visual  display  VFR  provided  G9  not  provided  CD 

IFR  only  CU 

(Brief  description  of  information  provided) 

Closed  circuit  color  T.V.  used  to  provide  realistic  visual  cues 
1500  ft  to  ground  accurate  14°  vertical  (downward)  cutoff  angle 
provided ,  but  lateral  view  restricted  to  40°  angle'.  Prominent 
horizon  provided.  Fairly  good  picture  resolution  enables 
3  ft/sec  touche!  -m  to  be  made  consistently,  but  not  good  enough i 
for  much  less. 

Task  (s) : 


Control  task  description:  a.  Visual  approach 

from  crosswi i 

id  leg 

7000-ft  altitude  at  recommended  approach  speed 

to  land  at  i 

-.OOP- 

elevation,  b.  IFR- ILS  ap\  roach,  capture  and 

track  3°  TLO 

from 

OM.  do  offset  crosswind  or  turbulence  2000  ft 

RVR.  c.  Pane  as 

b.,  except  with  20-k  erosswind,  light  turhuleii 

•  ,  mod  rate 

winds  hear  at  100  ft  above  ground.  2000  ft  PVR 

Auxiliary  tasks?  Yes  □  No  a 
Aircraft  environment: 

Day  [X]  N  j  ght  [X] 

Visibility  condit  ons:  Test  oariat  l.: 

Tu rbu  1  e nc e  [Xj  L t  gh  t  !  urbu  let.  *■ 

Kind  shear  IS  D  k  :  <  r  leP  ft  (3 

t r  o  s  s  w :  n  d  [X]  2  0  k 

Other  weather  conditions:  V 


k! 


4'' 

PILOT  EVALUATION  DATA 

Form  of  results 

Pilot  rating  ®  Comments  LJ  Questionnaire  QJ  Oral  debrief:  tg  IXl 
Other  CD 


Pilot  rating  scale:  Cooper- Harper _ _ _ _ 

P rinci pa  1  items  for  comment:  Airspeed  and  flight-path  control .  Pitch 

response.  'Pilot  workload. _ _ _ _ _ _ _ 

Program  assessment  : 


(Comments  and  recommendations  on  validity,  effectiveness,  etc. 
Confidence  factor  -  if  used) 


V\J 


APPENDIX  B  -  Example  2 

BRIEFING  GUIDE  AND  RATING  INFORMATION  FOR  HANDLING  QUALITIES  EXPERIMENTS 

PROGRAM  DEFINITION 

Ob j ectives 

General : 

Research  (X)  Development  lj  Acceptance  □  Other  □ 

Aircraft  role  Long-range  attack  bomber _ 

Flight  segment  of  interest  In-flight  refueling _ _ 

Parameter  or  variable  of  primary  interest  Lateral  control _ 

Mission  description: 

General  statement  of  the  required  operations  for  the  flight  segment 
of  interest  in  context  of  aircraft  role. 

1.  What  is  pi  lot- veh i cle  combination  required  to  accomplish? 

By  visual  references ,  attain  in-!  maintain  petition  yrlgt  j  ye  to 

tanker  within  presented  bounds  for  period  of  10  m:’n. _ 


What  are  the  conditions  under  which  the  e  required  operations  are 
to  De  carried  out  (i.e.,  aircraft  state,  environment,  and  cockpit 
i  ntert  ace) 
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Scope 

Short  look  (guide  to  further  tests)  1  I 
Long  look  (definitive  as  possible)  ® 

Task  related  only  (minimize  pilot  extrapolation)  I  ! 

Mission  or  flight  segment  related  (pilot  extrapolation  encouraged)  (Xl 
Nature  of  task  or  tasks  provided: 

Short  term  CJ  Long  term  [X| 

Critical  task(s)  included?  m  Or  left  for  pilot  extrapolation?  Q] 

Identify:  Succes s  or  failure  to  establish  and  mai n tain  position  with 

respect  to  tanker  with  light  turbulence  and  occassional  disturbance 

of  the  tanker, _ 

Are  simulated  disturbances  provided  or  to  be  supplied  by  pilot? 

Provi  Pwd _ _ 

Use  of  pi  1  ot  -  i  ni  t  i  a  t  ;>d  or  self-imposed  tasks,  disturbances,  or  short¬ 
term  maneuvers. 

Only  .luring  fgmj  liar.'  zation  or  rul'r<.’gamt  to  no ti fi cation  of  test 
7  2  t-O  . 

Provision  for  farm  liarization:  As  Jcsi  red  hj  pi  lot , 

Measured  per tormance  : 

Is  performance  to  he  measured? 

P  i  I  ot  veh  i  c  1  e  (  out  put  )  X 

Pi  lot  v\  ork  I  oad  (  i  nput  I  i 

Other  measurements'.' 
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Are  additional  runs  allowed  for  evaluation?  Yes 


explain: 


RATING  INFORMATION 


What  is  to  be  rated? 

(X)  Handling  qualities 
l-J  Task 


Other  Adequacy  of  simulation. 


52  Flight  segment  (flight  phase  or  subphase)  Special  phase  -  refueling . 
□  Composite  of  tasks  (specify  tasks  included  in  composite  rating) 


L-J  Composite  of  subphases  (specify  subphases) _ 

What  is  provided? 

Test  vehicle:  (aircraft,  spacecraft,  simulator,  etc.)  Simulator 


Aircraft  state: 

Normal  [X]  emergency  L.J  Identify  failure: 


Corf  i  gurat  ion  .  Sr  *' urh  n,; 


weight:  S00t  000  and  400 1  *0'0  lb 


‘la s '  d  •  t  r  i  but  i  on  :  Arry  n  'mite 
ai  ges  in  t  i  rc  ral  t  s  t  a  t  e  : 

i  Hi:  1  igur.it  i  on  changes  'dirirn  sk  or  designated  phase 
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I 


Transient  failures  (unanticipated  failures  to  be  introduced  into  task 
for  pilot  reaction  to  and  correction  of  resulting  disturbance) 

Yaw  damper  _  _ _ _ 


Cockpit  interface: 

(Brief  description  of  important  items  noting  unusual  or  detracting  or 
limiting  characteristics  in  particular) 


Principal  controls: 

Primary  controls:  Control  stick  with  light  control  forces s  good 
centering  and  low  friction.  Conventional  rudder  pedals  having 
light  forces  but  objectionably  poor  centering .  Requires  pilot 
adaptation  and  special  consideration. 

Secondary  controls  Trim  not  provi ded. _ _ 


Selectors:  Not  provided  -  configuration  unchanged. 


Cues  and  disturbances  provided: 


Motion:  l  itch,  roll,  and  yea r  (±45°).  Accelerations  attenuated,  by 
l  imi  Uti  tre.ns  .al  i  or  {  '  [>  ft  vertically ,  *10  ft  laterally ; 

’  10°  fore  (2nd  aft ) . 


V  i  sua  I  : 

! ns  t  ruments :  Convent i on a  1  XI  Move  1  1  ! 

lull  panel  i  Part  panel  L  1 

(Brief  statement  of  instruments  or  characteristics  which  may 
affect  the  evaluation) 


so 


External  vision: 


External  visual  display  VER  provided  (X)  not  provided 


I l-'R  only  f ~] 


(Brief  description  of  information  provided) 


Closed  circuit  T.V.,  enabling  visual  formation  flying  with 
stationary  model  of  tanker  aircraft. 


Task  (s)  : 


Control  task  description:  ;  1)  Close  on  tanker >  establish  contact, 
maintain  hookup  for  duration  o*'  re  ^uc  l  inn  operation  ?.>/  thout 

_ _ _ _ _ >. ...  _ t _ r _ >_ _ _ _ _ _ 

turbulence.  Came  as  ( 1 ),  except  for  simulated  light 

turbulence  and  occasional  a: -runt  li  srlacem-nt.  of  tanker  to 
simulate  dis-conne  ».* . 


Auxi 1 i arv  tasks?  Yes  j  ]  No  [X; 


A i re  ra i t  env i ronment : 


i.av  rx 


Ni  jjht 


V  i  s  i  b  i  1  i  t  v  cond  i  t  i  mis 


rU'Ci 


■  .  :  i  / 


''V'l 

I  in  !uil  once  ,-o-i 


Kind  shear 


i  rossw i nd 


( ) t  he  i  weather  condi*  >  oris 
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PILOT  EVALUATION  DATA 


Form  of  results 

Pilot  rating  (X)  Comments  Cl  Questionnaire  [X]  Oral  debriefing  CJ 
Other  CJ  Questionnaire  will  be  made  available  after  completion  of  tes t . 

Pilot  rating  scale:  Cooper-Harper _ _ 

Principal  items  for  comment:  Pilot  workload.  Influence  of  control _ 

system  chai'aat  eristics 3  if  any. _ 

Program  assessment: 

(Comments  and  recommendations  on  validity,  effectiveness,  etc. 

Confidence  factor  -  if  used) 

Real is m  of  tas k .  Adequacy  of  simulated  motion. _ 
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